Trigger workflows from APIs, on a schedule, or on demand. API calls are easy with authentication handled for you. Add durable delays that survive server restarts.

API Hero

A couple of years ago Github launched a neat “hidden” feature (seems like it’s still somewhat hidden these days) which allowed you to create a special kind of repository in your account. So you see, Github allows you to create this special repository and the README for this repo, acts as the landing page for your profile. You can read more about it in this by Github. official blog post You can share information about yourself with the community on GitHub.com by creating a profile README. GitHub shows your profile README at the top of your profile page. A Hello-World README page 👓 It’s pretty simple to whip up a bare-bones profile page. All you need to do is create this special repository named after your username and add a README page to it. Make sure to make this repository and for starters, go ahead and check the “Add a README file” box while creating the repository. public The newly created repo will contain a templated README courtesy of Github and Bob’s your uncle! Wow, this was pretty simple! 🍦 Well, yeah! But this is like a vanilla . It’s quick and it’s delicious. soft-serve But, that’s not what we really want now, is it? I for one want to have “some” sundae 😝 That’s not a typo, btw (Software Enginee 😅). That’s a dynamic piece of text. Lemme go over what all is there in and see if that is something that might interest you :) my Github profile Number of people who visited the profile. Profile Views — — Instagram, LinkedIn, Personal Website, Medium. Social badges — That’s a very simple text/banner generator that takes up any text you want and displays it in a typewriter fashion. Typewriter Text — A few bullet points about myself. Basic Information A daily programming quote that gets updated on the profile dynamically. Programming Quote of the day — — This article will also show up on the profile the next time you visit :) Latest Blog Posts Displays the song I’m listening to right now on Spotify! What’s Vibin? — — Most recent posts from an amazing Instagram account exploring Seattle, WA. Hey there, Seattle! — There’s a whole lot to unpack here so we’ll get to this later on in the article. Github Stats — That orange-colored card you see on the right. The Dev card The profile is not a static one but a dynamic one. Why is that? Because we use Github Actions — a whole lot of them — to update various parts of the README at different times of the day 🤘🏻. If this all sounds fun to you and you want to upgrade your Github profile as well, read on! Profile Views Counter 👁‍🗨 This one’s pretty simple to set up. All you need to do is, add the following line of code to your README wherever you want the counter to appear. ![](https://cdn.hackernoon.com/images/ckxz-5-f-75-v-00-z-00-as-638-qw-6-ofc.jpg) For more customizations to how this appears on your profile, refer to the official documentation . here Social Badges 🏅 For the badges, the was super handy. For ease of use, here are the relevant badge links from my profile. Feel free to change the “HANDLE” below and copy-paste the same to yours :) shields project We have the profile view counter and the badges set up. Woot Woot! Typewriter Text This . We’re not addressing the elephant in the room first! one’s also pretty simple to set up Head over to the website linked above and add whatever text you need along with your own customizations and finally, copy the markdown link for the same and add to your README. Easy peasy! Basic Information Nothing’s really fancy here. It’s just some pieces of information about myself. You don’t necessarily have to write about your technical accomplishments. Go crazy with what you write. I added my tennis profile, my pronouns, my hiking interests, in addition to where I work. The world is your oyster! In general, though, try to keep it anywhere from 8–10 bullet points. Any more than that and it will start to weigh on the reader. Programming Quote of the day 📜 There are multiple ways of adding a daily quote to your profile. If you’re feeling a little adventurous, you can write a script to call an external API that fetches the most recent quotes. The API being: https://programming-quotes-api.herokuapp.com/Quotes?count=2 And this is what the response looks like in the JSON format. You can then parse this response to fetch the quotes, however many you requested — count=2 in the API — and display them in the README in some format. In order to update the quotes regularly, you’ll need to add a Github Action that will run your script on a regular basis, maybe once each day, and update the README with the quotes. While this might seem the programmatically pleasing way to go, the final result might require a lot more effort as the presentation is equally important for the quotes. This seems more aesthetically pleasing than the quote itself, doesn’t it? If you think it does, then go ahead and follow the details in this . great repository Essentially, you need to add this image tag to your README file wherever you want the quote to appear. A new quote might appear whenever someone opens your profile page. This is the one-liner solution that in most likeliness, does the API calls and SVG rendering in the background and simply returns an image. Notice the “ ”. Well, quotes-github-readme.vercel.app that’s the hosted service linked to the Github repository that is running the quote image generator server or something. Pretty cool stuff! Daily Dev Card 🏋 Do you like to read? If the answer’s yes and your reading list includes some technical blogs from time to time, you might want to consider daily dot dev. Personally, I’ve added the chrome extension to get a regular feed of articles about topics I love to read about. Not really publicizing the platform but it is a great extension and platform. They generate an awesome dev-card for you based on your reading interests including all the badges, publications, topics you like, etc. The more you read, the better it gets. Follow this on how to add this dev-card to your profile. Most importantly, this is a dynamic card i.e. we use Github Actions workflow to update the card at frequent intervals. great blog post Latest Blog Posts ✏️ This one’s especially important if you’re an avid writer. There are a lot of platforms these days for writing and sharing both technical and non-technical content. There’s , , , (of course), and maybe your own personal website. Medium Dev.to Ghost.io hackernoon.com Basically, if you have any kind of RSS feed, you can hook it up with the README profile and keep updating it with the latest posts. How, you might ask? Via Github Actions of course :) This Github Action periodically replaces the following comments in your README with the hyperlinks to your recent 5 blog posts. Follow the steps mentioned to set this up. here Also, as a general rule of thumb, if you end up using any open source software, make sure to start it and share it! Goes a long way to help open-source developers. What’s Vibin? 🎧 🎼 I had the most fun setting this up. Before we get to the steps, one might ask why do we need Spotify’s now playing on a Github profile page? Well, why not? A developer is not just a bunch of commits, PRs, and open source projects. This is just an effort to make it more personal. Go ahead and add those Strava runs, Goodreads reading lists, anything you want really. This contains all the details on how you can set this up for your README. great repository I ran into trouble setting this up with Vercel or Heroku based on the steps . here The server is simply running the script “fetch_spotify.py” to get the currently playing song details for a user and then render an SVG based on a template. Instead of a hosted service, I decided to make use of Github Actions to call the same script every 5 minutes (can’t have a higher frequency) and do the exact same thing: generate an SVG from the currently playing song. Here are the steps I followed to set this up: Generated the Client Id, Client Secret, and the Refresh token as mentioned in the repository linked above. Copied the templates folder, , and templates.json spotify.py Renamed to fetch_spotify.py and made some changes to it. We don't want to run a flask server. Instead, it should be invokable via the main function. spotify.py Setup a new Github Actions workflow to call the script at a set schedule The script runs every 5 minutes and generates the “spotify.svg” file which gets placed in the repository root folder. From there, it gets picked up by the README file. Hey there, Seattle! 🏞 This might be a bit too much on the Github profile, but, I felt it’s worth showing off the beautiful place that Seattle is which is where work and home-away-from-home is currently for me :) For this, I wrote a Python script that uses the module to download the recent posts from a given Instagram account. We don’t need any sort of login functionality for accessing public accounts so make sure the account you choose is public. instaloader Again, the script is attached to a Github actions workflow so that we keep the profile updated with the latest Instagram posts from the account. You just need to set “INSTAGRAM_PUBLIC_HANDLE” as the secret for your Github Actions workflow to access and that’s about it :) The script looks something like this: The script downloads the recent 3 posts from the Instagram account and stores them in separate directories. Then, the README file can access them and display side-by-side. Github Stats 🍬 🍭 This is really the meat (vegan!) of the profile. It’s the place where you really show off what your open-source contributions amount to in a crisp, easy-to-digest fashion. Mine doesn’t look like much right now but it’s a work-in-progress. The first thing we have here is the Github stats card. This is a purview of all of your contributions so far on the platform like total stars, total PRs, commits, etc. Follow the steps mentioned to get the design you want for your card. here Next up, we have the top-languages card. This is something we tend to mention in our resumes. However, it’s better if Github compiles that for us based on our LOCs, right? From the same repository as before, to incorporate the top languages card in your profile. follow the steps Finally, there’s the big Github stats image that has a bunch of theatrical statistical stuff from your profile and contributions over the years. That comes from a separate repository and like all of these Github stats, that is also something that is dynamically updated. The stats and top languages above come from a hosted service, but, the image/card is updated by Github Actions. There’s a whole lot of customization around what you want and don’t want to include in the card and you can refer to the information mentioned for that. here Personally, I got a lot of inspiration for my profile from Waren Gonzaga. I also kinda copied the Github Actions for generating the stats card from their repository as well. Other Notable mentions That’s pretty much what all was needed to set up a great (rockstar? you tell me!) Github profile. There are a few other notable mentions that I’d like to point out though. is a great tool for beautifying your code snippets and adding them to your articles. The only downside is, they’re images and not scrollable like Github Gists or something. Carbon Next up me of the greatest, inspiring Github profiles. This is where I personally started to get some inspiration for my profile. is a brilliant collection for so is a great place to find some fun and interesting animated emojis that you can add to your profile. Slackmojis Finally, the footer for my profile README is this wave-like animation that keeps changing colors and gradients on its own. If this is something that interests you, head over to the link shared before ( . capsule-render) Thanks! If you’ve made it this far in the article, cheers to you! Thanks a bunch for reading :) to my repository. Feel free to fork and play around. Here’s the link

How to Make a Rockstar Github Profile README

Where do you keep your dad jokes? In a of course! Let's imagine that you are a site maintainer for the world's best dad joke database. Your app communicates with the database using a REST API that allows you to retrieve jokes and post ratings for those jokes. Visitors to your site can rate each joke they see via a simple user interface. dadabase Recently you heard of a fancy new technology called GraphQL that provides the flexibility to request only the data that you need using a single API endpoint. It sounds neat, and you'd like to start using it in your app. But, you'd really prefer not to make any breaking changes to the existing REST API. Is it possible to support both the REST API and the GraphQL API in your app? You're about to find out! In this article we'll explore what it takes to implement a GraphQL API on top of an existing REST API. This strategy allows you to start using GraphQL in legacy portions of your app without breaking any existing contracts with functionality that may still rely on the original REST API. If you'd like to see the end result, you can find the and the . Don't forget to as well to groan at some jokes. code for the REST API here code for the frontend and GraphQL API here visit the app The Initial Architecture The app's backend was originally built using and . JSON Server utilizes to provide a full REST API to a mock database generated from a simple JSON file. A separate Express server takes care of serving the static HTML, CSS, and JavaScript assets for the frontend. The frontend is implemented in vanilla JS and uses the browser's built-in to make the API requests. The app is hosted on to make deployment and monitoring a breeze. Node JSON Server Express Fetch API Heroku Our JSON file contains information for a few jokes as well as some ratings. It's reproduced in full below: JSON Server takes that file as a starting point for the database and then implements a REST API that includes support for GET, POST, PUT, PATCH, and DELETE requests. The magic of JSON Server is that using this API really does modify the underlying JSON file, so the database is fully interactive. JSON Server can be started directly from an npm script without any additional setup, but in order to provide a little more configuration and a dynamic port, we can instead write a few lines of code like so: You can test out our mock database by cloning the , running , and then running . If you navigate to you'll see all of the jokes. Navigating to will display all the ratings. repo for the API npm install npm start http://localhost:3000/jokes http://localhost:3000/ratings /jokes API endpoint returns all the jokes when running the app locally Wonderful! We can run our app's backend locally in the browser. Now let's get our API hosted on Heroku. First, we need to . After that, we can log in, create the app, push it to Heroku, and open the new app in our browser in four easy steps: install the Heroku CLI heroku login heroku create dad-joke-dadabase-rest-api git push heroku master heroku open # log in to your Heroku account # create the Heroku app # deploy the code to Heroku # open the Heroku app on your machine And look, now we have a publicly available API out on the web! /jokes API endpoint returns all the jokes when hosting the API on Heroku Building the User Interface Now that we have a working REST API, we can build the frontend to consume that API and display the user interface for viewing and rating jokes. The HTML provides a shell of the page with containers into which the JavaScript will insert content for each joke. The JavaScript is shown below. The key pieces that interact with the REST API are the two fetch requests. The first fetches all of the jokes from the database by hitting the endpoint. The second makes a POST request to the endpoint to submit a new rating for each joke you rate. /jokes?_embed=ratings /ratings Dad joke "dadabase" user interface allows you to rate each joke Setting Up Apollo Server So, that's the existing app architecture: a simple frontend that interacts with the database via a REST API. Now how can we begin using GraphQL? We'll start by installing , which is a package that allows us to use with Express. We'll also install the package to help us integrate the REST API with Apollo Server. Then we'll configure the server by writing the following code: apollo-server-express Apollo Server apollo-datasource-rest As you can see, we configure Apollo Server with type definitions ( ), , and . The contain the for our GraphQL API. In it, we'll define types for our jokes and ratings as well as how to query and mutate them. The tell the server how to handle various queries and mutations and how those link to our . And finally, the outline how the GraphQL API relates to the REST API. typeDefs resolvers dataSources typeDefs schema resolvers data sources dataSources Here are the type definitions for the and types and how to query and mutate them: Joke Rating The jokes data source defines methods for calling the original REST API endpoint to create, read, update, and delete jokes from the database: The ratings data source looks nearly identical, but with "rating" substituted for "joke" in every instance. ( if you'd like to see the code for this.) Refer to the GitHub repo Finally, we set up our resolvers to show how to use the data sources: With that, we have everything in place we need in order to start using our GraphQL API through Apollo Server. To get our new frontend and GraphQL API hosted on Heroku, we'll create and deploy a second app like so: heroku create dad-joke-dadabase git push heroku master heroku open # create the Heroku app # deploy the code to Heroku # open the Heroku app on your machine Replacing the Endpoint to Fetch Jokes You'll recall that we have two endpoints used by the frontend: one to fetch jokes and one to post ratings. Let's swap out the REST API for our GraphQL API when we fetch the jokes. The code previously looked like this: Now to use the GraphQL endpoint, we can write this instead: We can run the app locally now and verify that the user experience still works properly. In fact, from the user's point of view, nothing has changed at all. But if you look at the network requests in your browser's developer tools, you'll see that we're now fetching our jokes from the endpoint. Amazing! /graphql The Network tab shows a request is being made to the /graphql endpoint now Replacing the Endpoint to Submit Ratings One API down, one to go! Let's swap out the ratings submission functionality now. The code to post a new joke rating previously looked like this: To use our GraphQL API, we'll now use the following: A quick test gives us some promising results. Once again, the user experience remains unchanged, but now we're fully using the endpoint for both our requests! /graphql Conclusion We did it! We successfully wrote a GraphQL API endpoint on top of an existing REST API. This allows us to use GraphQL to our heart's content without breaking existing functionality and without modifying the original REST API. Now we can deprecate the REST API or get rid of it completely at a later date. While our dad joke database is entirely fictional, nearly every technology company that existed prior to GraphQL's release in 2015 will find themselves in this same position of migrating to GraphQL if and when they choose to do so. The good news is that Apollo Server is flexible enough to pull data from a variety of sources, including existing REST API endpoints. Previously published at https://codeburst.io/how-to-implement-a-graphql-api-on-top-of-an-existing-rest-api-db8b343ddb5a

How To Connect GraphQL On Top of Existing REST API

So, you’ve chosen the programming language to learn and decided to start browsing all over the Internet looking for a course that will satisfy your requirements. That’s where you may encounter a problem because there are tons of great resources on the web to help newbies learn coding from nothing. The question is: which one to choose? I’ve put together the top 16 websites I find the best for novices. To begin, I recommend taking a quick look at the chart with pricing and other useful information about each platform. 1. Udacity is an appropriate solution for everyone who desires to teach themselves to code yet doesn’t know how to learn programming and has a tight budget. Fortunately, lots of free separate classes — comprising video lectures, exercises, quizzes, and other tasks for aspiring users — will let you start with the fundamentals of computer programming. If any issues occur with task accomplishment, feel free to seek help from fellow developers within the forum. Udacity Next to free classes, Udacity offers Nanodegree training focused on teaching you specific skills so you can land a job in industries like machine learning, data science, Android and iOS development, etc. Elaborated with many industry leaders, Nanodegree training ensures support from professional coaches to make sure you acquire the necessary skills. 2. CodeGym The motto sounds like, “Everyone can become a programmer! It’s all down to the right approach to learning,” and they seem to find this the right approach. CodeGym is an online platform dedicated to learning the Java coding language. Its programming course consists of 80% practice, offering more than 1,200 tasks you should accomplish to upgrade your skills. CodeGym You can also join the forum to infuse yourself into a large and friendly community. The students share their achievements in learning Java, respond to each other’s queries, and talk over complex problems they experience during the learning process. Among other things, the company uses technologies of gamification to make the educational process exciting. You train with a character that needs to be upgraded to open the next level and keep on course. Learning Java through the game-based approach is quite an entertaining process, so you don’t stand a chance of wanting to give it up. 3. Treehouse Like many other learning websites, is dedicated to teaching you skills you may need to implement various designs, ranging from apps to WP blogs. Training materials presented in over a thousand videos are shortened and practice-oriented, so the learning process will not take you forever. To put some sharper points on the material, students get to complete quizzes and other code challenges. Treehouse For every single completed course, future coders receive a badge that demonstrates their level and is visible to everyone on the platform. This is a chance to be seen by companies looking for specialists to join their projects. 4. freeCodeCamp I found distinctive. Next to a big community accompanied by a whole lot of guides, the resource boasts an unsurpassed blog that likely has no analogs among similar platforms. However, I would not recommend it to beginners who are not well versed in programming theory — you may have a hard time handling practical and project-based tasks here. freeCodeCamp 5. Codecademy There may be no one who’s never heard of at least once. It is a sought-after education resource with multiple courses and tasks created to take you to a higher level. When choosing this project, you get access to over 300 hours of useful code-related content available to people with different levels of training. Besides, a well-structured program will let you create an education schedule based on your workload and follow it. Codecademy 6. Khan Academy positions itself as a noncommercial platform aimed at making education available for everyone. It probably won’t give you many practical skills, but you will leave with an understanding of computer science fundamentals. After finding out how organizing data in a computer works, mastering object-oriented programming concepts, and starting to think like a coder, you can go for or for more practice. Khan Academy CodeGym freeCodeCamp 7. Code4Startup If you’ve already mastered programming and strive to write your first line of code, consider . This project ensures a win-win partnership for both novice coders and startups. You write code for existing businesses, while the owners can save on headhunting. Besides, mastering your skills this way allows for establishing a promising portfolio and running a job search. Code4Startup Accompanying the practical classes are many free theoretical classes, such as Wistia API, Heroku, Learn Ruby on Rails, etc. 8. One Month Are you looking to challenge yourself and acquire the fundamentals of JS, Python, and Ruby quickly? Then is good to go. It is an intensive program specialized at a fast learning speed — the course lasts 30 days and lets you move on to build a brilliant career. One Month Most classes charge you a fee, but there are still a few free ones, including blog posts and podcasts that let you keep up with the latest coding-related news without spending a penny. 9. The Odin Project The , a beginner-oriented course, never leaves you alone after you finish a program. Its project-based curriculum lets you start building projects for your portfolio during your training and get a better understanding of what you can use your acquired skills for. Besides, the platform has built a strong community around itself so that students can rely on other specialists who can help them through the process. Odin Project 10. Codewars Let’s face it, programming is sometimes dull and not always that simple. So many of you might have lost motivation and even gave up training at least once. We are all in the same boat; let’s be honest. But is a project aimed at proving that programming can be fun. It contains lots of engaging assignments to let you test your developer skills while competing with colleagues. Codewars The Codewars project is also known for its solid-together community. If you have a question, feel free to seek help from the developers — they will give a comprehensive response that is more likely to affect your further advance. 11. Sitepoint is a vast library of books on frameworks, languages, and projects accompanied by guides dedicated to web development. If you want to find answers to some of your questions, use the search bar on the homepage. After entering the query, you’ll get a massive amount of relevant educational material. With a handy filter, you can then sort the articles by categories or authors, depending on what you are looking for. Sitepoint 12. TutsPlus , offering in-depth training materials, will help you level up from novice to pro coder shortly. With numerous articles, videos, and tutorials covering web design and programming, you can start learning to code from the very beginning. Plenty of free guides will let you move from basic to advanced level at a speed that is comfortable for you. TutsPlus If, after completing free classes, you want to keep learning with TutsPlus, apply for a premium plan. There are many paid courses, e-books, and other premium content worth investing in. 13. W3Schools will come in handy for both newbies and expert developers. It’s a highly used reference platform with a large number of training aids on most popular coding languages. The project strives for simplicity and straightforward learning, providing detailed instructions on how to use the code. To start working with this platform, you can pass a short test that will show your knowledge level and help you choose the most fitting program. W3Schools 14. GeeksforGeeks The website is one more wholehearted recommendation for those looking for useful articles dedicated to programming languages. The platform will give you access to all kinds of training materials, from basic to premium courses, from programming problems to practice to entrance exams, etc. I hope you will relish the platform just like I do because it demonstrates an inclusive approach to every technical concept and different ways to resolve programming issues to let you pick the most appropriate solution. GeeksforGeeks 15. Code Avengers will be suitable for entry-level users rather than experienced developers. The curriculum keeps 1K+ engaging quizzes on JS, Python, HTML, and CSS that will ensure more practice and give students an idea of how to use the acquired skills. You will start by learning how to create minimalist websites and apps and step up to building full-fledged web designs that can be shown to potential employers. Code Avengers 16. SoloLearn is a perfect solution for everyone who’s always on the move. The resource has mobile apps to let you learn to code wherever you are. Given a set task, you can go with web development and learn HTML, CSS, JS, jQuery, PHP, and SQL or choose back-end and software development with Python, Ruby, Java, Swift, C++, or C#. SoloLearn SoloLearn is not only an online learning platform; it is also a strong network of specialists who can help you throughout your training. Also published here . Wrap Up A comprehensive educational platform overview should make finding the best way to learn programming much easier. Depending on your goals, you can choose from a theoretical plan and, as you increase your basic knowledge, move on to more practical classes. So, if you are ready to obtain new skills in web development, go for it without delay.

Best 16 Websites to Learn Programming in 2020

I would like to share a recent with bug bounties. I’ve been doing bug bounties for years and I just now am trying-out the big boys on the playground — and — and the experience is… well, different. experience bugcrowd.com hackerone.com The incentive is there, I guess, to be able to negotiate with someone which cares on delivery times and holding accountability against the firms you are assessing, but I never felt the need of involving a third party in the process, maybe because I’m not making a living out of it and doing it out of fun and genuine care since usually I’ve been discovering vulnerabilities on websites I’m already an active user on. The extra buck, T-shirt or mere mention on a Hall-of-Fame of some sort, is mostly for bragging rights. I’ll come back to some of my own two-cent level insights at the end of this post — but first, let’s go on and tell you about the vulnerability found on Heroku. What’s Heroku? is pretty amazing, actually. They have an integrated PaaS solution that allows developers host their apps on the cloud with maximum ease. Heroku Large as well as small companies are using Heroku non-stop, so I guessed that’s a good place for interesting vulnerabilities that might affect them all by spotting an organization-indifferent vulnerability. Login API call, Spot the Response Anomaly Heroku has several options of interfacing with their service. Web-based is pretty common usage and API follows. There is a Heroku Command Line Interface (CLI) tool which basically is a wrapper to API calls. I used the CLI tool in order to contact the site and started on messing with some inputs and after a time I focused on the login mechanism which looked pretty simple. It is based on a POST request, accompanied by and parameters. username password At first, when messing with the inputs I figured nothing really matters, I was hoping for an error output or any kind of response changes between real usernames against bogus ones. Even though I didn’t see any of those error text changes — there was one more subtle change in response. A login request looks something like that: POST HTTP/1.1Host: api.heroku.comUser-Agent: heroku-cli/5.2.39-010a227 (windows-386) go1.6.2Content-Length: 54Accept: application/jsonContent-Type: application/x-www-form-urlencodedAccept-Encoding: gzip https://api.heroku.com/login password=fjfjfjfj&username=bogus%40gmail.com And an appropriate response would be: HTTP/1.1 404 Not FoundCache-Control: private, no-cacheContent-Type: text/html;charset=utf-8Date: Mon, 29 Aug 2016 16:49:52 GMTRequest-Id: b3ca4a2c-1adf-4244-4cb0-17fbbc9deb06Server: nginx/1.8.1Strict-Transport-Security: max-age=31536000; includeSubDomainsVary: Accept-EncodingX-Content-Type-Options: nosniffX-Frame-Options: SAMEORIGINX-Runtime: 0.018023806X-Xss-Protection: 1; mode=blockContent-Length: 0Connection: keep-alive So, as said, nothing more than a 404, but on the other hand it is somewhat generous on it’s optional headers — “so what?” you might ask. That makes the whole difference. To be specific, the thing that caught my eye was the “ header. X-Runtime” From it’s name I could only guess what it is that I found. But it needed a bit more probing to be sure about it. Freeze, you bastard! First stop is looking at the 1,000,000th time at the HTTP RFCs, noting that there is nothing like it in there — so it must be something added by another component on the web server. A short scan revealed it is most probably derived by a server side component relying on Ruby/Rails. One of those projects, named , was first to come up in search, that is to say that we can’t be sure of whether that’s the Ruby code that is operating on the site, but we’ll take it as a test case which proved to be sufficient at the end. Rack For those of you that never heard of it (like myself), it’s GitHub page reads: Rack provides a minimal, modular, and adaptable interface for developing web applications in Ruby. By wrapping HTTP requests and responses in the simplest way possible, it unifies and distills the API for web servers, web frameworks, and software in between (the so-called middleware) into a single method call. Peering into it’s source code, the aberrant header is explained within : lib/rack/runtime.rb Now we can be sufficiently sure of our common-knowledge understanding of ‘X-Runtime’ meaning. It is referring to the time it took the operation to complete. Are you thinking of what I am thinking? To confirm this thought we have to get a larger test sample set against different mix of non-existent and existent user names on the system. And take note of timing differences and if there any distinguishable kind of delta. Existent user names inclusion yeilded the following timing annotation: X-Runtime: 0.093990008X-Runtime: 0.088857339X-Runtime: 0.09251876X-Runtime: 0.089843447X-Runtime: 0.091845765X-Runtime: 0.101260549 And when using non-existent user names (I’ve used the correct scheme of e-mail address, just to be safe): X-Runtime: 0.027787351X-Runtime: 0.018023806X-Runtime: 0.020997733X-Runtime: 0.021645124X-Runtime: 0.016645836X-Runtime: 0.021828831X-Runtime: 0.022211143 Old-School Side Channel Timing-Attack — ❤ All of this assessment is funneled into one final conclusion — There is a clear distinctive (~1 order-of-magnitude) difference in times between bogus and true user names on the system. Why is that even possible? Probably because of cryptography-related operations involved in the process of checking the password against it’s true hashed password. It is a by-the-book example of a Timing Attack. Once the server receives the parameters from the user it first checks the user to see if it is relevant, if it isn’t the operation comes to an end and respond the to the user. On the other case, if the user name is found to be relevant — it progresses forward-on to check the password given by the user against it’s local hashed copy of the password, which is a very intensive operation in contrast to the short trip to username-list that it followed. Because of that difference in operation time, we can perceive these distinctive routes. 404 Many applications are doing the same, but mostly because of latency it is somewhat harder to determine a definite conclusion upon a timing-based attack. In this case — the Ruby server is giving us all we need to get to this conclusion — all with one simple POST request. Cool! How to overcome the attack? This can be tackled on many levels. Since I’m not aware of Heroku’s considerations on keeping this header on, I will try to cover different approaches as there can be practical limitations of nearly each one of those. this is the most obvious recommendation as that’s the fault that led us to our conclusion, but this is just the end-result of something much inherent — the times computation might be observed on other, less straight-forward ways as the time of delay of the response. What I observed by counting round-trip time from sending a request until receiving a response was not something that is distinguishable by that alone. Runtime value can be used for debugging purposes and I have met with clients in the past that couldn’t just remove debug headers from all or even some subset of responses-if that’s the case they might modify it on a higher web-server level and make an effort to delete the generous line before compiling the final response. Very ugly patch indeed, but practicality overpower elegance sometimes. Remove ‘X-Runtime’ header from responses — relying on the assumption that Heroku has granular control over the process of user authentication, they might delay the process of checking username and password parameters to an extent of time. This wouldn’t necessarily be projected through X-Runtime if they decide on leaving that in place. Make all operation times equal — injecting random length delays into the process can, in effect, make those times erratic. The problem with this approach that on the greater scale in multiple checks over each guess you will get (assuming normal distribution of random-length delays) a central tendency towards a larger result. This, too, doesn’t necessarily be projected through X-Runtime. Add random timings to operations — This is considered a solution in case you are seeing this attack as something that is threatening a customer base as a whole and not as a targeted attack against a small subset of users. By this definition — you should be able to limit the amount of requests from a single source on a given time, this makes the efforts of the less-determined attacker redundant. This is not a bulletproof solution in any case since this usually circumvented by attackers which are using VPN, Anonymization network or a botnet to cover their tracks. Limit the amount of requests — Heroku/BugCrowd’s Response According to Bugcrowd’s terms — User Enumeration vulnerabilities are not in-scope, and they actually penalized me for that in -1 points (honestly, I should have known better. To my defense it wasn’t clear to me that there is a separate terms not only from the vendor but from BugCrowd as well that need to be adhered.) The thing is, as I mentioned before — that wasn’t my point, as many of my peers I wasn’t doing this for profit, I was trying to make a site I evaluated for using for my own needs and if happen to be, as in this case, prompt for vulnerabilities. Whether I get the bounty is the secondary objective, the thing that worried me is that because BugCrowd consider it out-of-scope doesn’t make this vulnerability something we shouldn’t discuss. The terms should be there to be defined that not everything is going to make you a small buck on the side, but once you find something that is out of scope for bugcrowd why should it be considered out-of-scope of mitigation for the vendor? security I truly don’t have anything against Heroku nor BugCrowd, they both are comprised of great, passionate people from my experience, but bureaucracy in this case contradicts progress and I wish those unwritten procedures should be taken care of for the benefit of our community.

“The Giving Ruby”—The Strange Case of User Enumeration on Heroku (Not Fixed)

The , illustrated below, can be applied to most aspects of technology: Gartner hype cycle As new innovations enter their respective cycles, expectations are eventually realized—leading to some level of adoption. The goal for every innovation is to reach the plateau of productivity where consumers have determined that the reward of adopting the innovation far outweighs any known risks. At the same time, there is a point where the plateau of productivity begins to diminish, leading to an exodus away from that innovation. One simple example would be (or beepers), which were common before mobile phones/devices reached the plateau of productivity. pagers As technologists, we strive to deliver features, frameworks, products, or services that increase the plateau of productivity. The same holds true for the ones that we use. Recently, I felt like my current hosting platform began falling off the plateau of productivity. In fact, a made me wonder if it was time to consider other options. recent announcement Since I had a positive experience using the PaaS, I wanted to look at how easily I could convert one of my Heroku applications, adopt PostgreSQL, and migrate to Render. I’m describing that journey in this two-part series: Render Part 1: We’ll focus on migrating my backend services (Spring Boot and ClearDB MySQL). Part 2: We’ll focus on porting and migrating my frontend Angular client. Why I Chose Render If you have never heard of Render before, check out some of my previous publications: Spend Zero Time on DevOps with Render PaaS The Perfect Cloud: AWS, GCP and Azure all-in-one Purpose-Driven Microservice Design Launch Your Startup Idea in a Day How I Used Render To Scale My Microservices App With Ease What I find exciting about Render is that they continue to climb the slope of enlightenment while actively providing a solid solution for adopters recognizing the plateau of productivity. As I’ve noted in my articles, Render offers a “Zero DevOps” promise. This perfectly aligns with my needs since I don’t have the time to focus on DevOps tasks. The Heroku platform has several things that I am not too fond of: Daily restarts led to unexpected downtime for one of my services. Entry-level (all I really need) Postgres on Heroku allows for four hours of downtime per month. Pricing levels, from a consumer perspective, don’t scale well. From a pricing perspective, I am expecting to see significant cost savings after migrating all of my applications and services from Heroku to Render. What is more amazing is that I am getting better memory and CPU for that price, with linear scaling as my application footprint needs to grow. Converting a Single Service As I noted above, this is part one of a two-part series, and I’ll focus on the service tier in this article. The service I want to convert has the following attributes: Spring Boot RESTful API Service Heroku CloudAMQP (RabbitMQ) Message Broker Heroku ClearDB (MySQL) Database (single schema) Okta Integration On the Render PaaS side, the new service design will look like this: Render Web Service hosting Spring Boot RESTful API Service (via Docker) Render Private Service hosting RabbitMQ Message Broker (via Docker) Render Postgres with the ability for multiple schemas to exist Okta Integration Below is a side-by-side comparison of the two ecosystems: My high-level plan of attack for the conversion is as follows: Prepare Heroku for Conversion Before getting started, it is recommended to put all existing Heroku services into . This will prohibit any consumers from accessing the applications or services. Maintenance Mode While the source code should already be backed up and stored in a git-based repository, it is a good idea to make sure a database backup has been successfully created. Conversion of Heroku Services From a conversion perspective, I had two items to convert: the service itself and the ClearDB (MySQL) database. The conversion of my Spring Boot RESTful service did not involve much work. In fact, I was able to leverage the approach I used for a of mine. previous project For the database, I needed to convert from MySQL to PostgreSQL. My goal was to use Render’s to easily migrate Heroku Postgres to Render Postgres, but I needed to convert from MySQL to PostgreSQL first. Heroku Migrator Initially, I started down the path, which seemed to be a common approach for the database conversion. However, using my M1-chip MacBook Pro led to some unexpected issues. pgloader Instead, I opted to use to convert MySQL to PostgreSQL. For more information, please check out the “ ” section below. NMIG Highlights From the Database Conversion Create Services in Render After converting the database and the Spring Boot RESTful service running inside Docker, the next step was to create a Render Web Service for the Spring Boot RESTful API service. This was as easy as creating the service, giving it a name, and pointing to the appropriate repository for my code in GitLab. Since I also needed a RabbitMQ service, I followed to create a RabbitMQ Private Service running on Render. This included establishing a small amount of disk storage to persist messages that have not been processed. these instructions Finally, I created the necessary in the Render Dashboard for both the Spring Boot RESTful API service and the RabbitMQ message broker. environment variables Initialize and Validate the Services The next step was to start my services. Once they were running and the APIs were validated using my Postman collection, I updated my client application to point to the new Render service location. Once everything was up and running, my Render Dashboard appeared as shown below: Next Steps All that remained at this point was to delete the databases still running on Heroku and remove the migrated services from the Heroku ecosystem. When using Heroku, any time I merged code into the master branch of my service repository, the code was automatically deployed, provided I used in my source repository. GitLab CI/CD to deploy to Heroku However, there is no need to add code to the source file repository with Render. I simply needed to specify the Build & Deploy Branch in the Render Dashboard for the service: I love the Zero DevOps promise. Highlights From the Database Conversion By following the steps above, the conversion from Heroku to Render was smooth and successful. The biggest challenge for me was the conversion of data. At a high level, this mostly boiled down to a series of commands executed from the terminal of my MacBook Pro. First, I started a local Postgres instance via Docker: docker run --publish 127.0.0.1:5432:5432 --name postgres -e POSTGRES_PASSWORD=dbo -d postgres Next, I created a database called “example” using the following command (or pgAdmin): createdb example For converting my ClearDB (MYSQL) instance running on Heroku to my example Postgres database running locally, I used , which is a Node.js-based database conversion utility. NMIG After installing NMIG, I set up the config.json file with database endpoint information and credentials, and then I ran: /path/to/nmig$ npm start Next, I backed up the data to a file using the following command: pg_dump -Fc --no-acl --no-owner -h localhost -U postgres example > example.dump Rather than go through the hassle of creating a signed URL in AWS, I just used the client to import the backup into a newly created Postgres instance on Heroku. pgAdmin With the Postgres instance running and data validated, I on the Render PaaS. Then all I had to do was issue the following command: created a new Postgres database pg_restore --verbose  --no-acl --no-owner 
-d postgres://username:password@hostname.zone-postgres.render.com/example example.dump Lessons Learned Along the Way Looking back on my conversion from Heroku to Render, here are some lessons I learned along the way: I had a minor issue with the Postgres database updating the date/time value to include the DST offset. This may have been an issue with my original database design, but I wanted to pass this along. In my case, the impacted column is only used for Date values, which did not change for me. I included a database column named END in one of my tables, which caused a problem when either Postgres or Hibernate attempted to return a native query. The service saw the END column name and injected it as a SQL keyword. I simply renamed the column to fix this issue, which I should have known not to do in the first place. With Render, I needed to make the RabbitMQ service a Private Service because the Web Service option does not expose the expected port. However, with this approach, I lost the ability to access the RabbitMQ admin interface, since Private Services are not exposed externally. It looks like Render plans to address this . feature request All in all, these minor hurdles weren’t significant enough to impact my decision to migrate to Render. Conclusion The most important aspect of Gartner’s plateau of productivity is providing products, frameworks, or services that allow consumers to thrive and meet their goals. The plateau of productivity is not intended to be flashy or fashionable — in a metaphorical sense. When I shared this conclusion with Ed, a Developer Advocate at Render, his response was something I wanted to share: “Render is pretty avowedly not trying to be ‘fashionable.’ We’re trying to be unsurprising and reliable.” Ed’s response resonated deeply with me and reminded me of a time when my told me my code came across as “boring” to him. His comment turned out to be the greatest compliment I could have received. You can read more . former colleague here In any aspect of technology, the decision on which provider to select should always match your technology position. If you are unsure, the Gartner hype cycle is a great reference point, and you can get started with a subscription to their service . here I have been focused on the following mission statement, which I feel can apply to any IT professional: “Focus your time on delivering features/functionality that extends the value of your intellectual property. Leverage frameworks, products, and services for everything else.” - J. Vester When I look at the Render PaaS ecosystem, I see a solution that adheres to my mission statement while residing within my hype cycle preference. What makes things better is that I fully expect to see a 44% savings in my personal out-of-pocket costs — even more as my services need to scale vertically. For those considering hosting solutions, I recommend adding Render to the list of providers for review and analysis. You can get started for free by following . this link The second part of this series will be exciting. I will demonstrate how to navigate away from paying for my static client written in Angular and take advantage of Render’s free Static Sites service using either Vue or Svelte. Which framework will I choose … and why? Have a really great day!

Time For Me To Fly… To Render

আমার জন্য সময় উড়তে ... রেন্ডার

Zeit für mich zu fliegen ... zu rendern

Hora de volar… de renderizar

Il est temps pour moi de voler… de rendre

टाइम फॉर मी टू फ्लाई... रेंडर करने के लिए

私が飛ぶ時間…レンダリングする時間

날아갈 시간… 렌더링

Hora de eu voar… de renderizar

Benim İçin Uçma Zamanı... Render Yapma Zamanı

是时候让我飞了……渲染

Hello world! In preparation of a talk for an in Hamburg I struggled and stumbled upon some documentations and tutorials for using Bitbucket and Heroku together. So I’ve decided to write my own first developer article about Atlassian Community Event Building a basic CI/CD pipeline for deploying a to using React App Heroku Bitbucket Getting Started Step 1 — Create React app In this tutorial we will focus on deploying a React app to Heroku, so we will just use the React starter app. To run the App in Heroku we’ll add a basic express server setup later on. Prerequisite for creating and running a React app make sure you have: 1. and installed Node.js npm 2. installed (optional) Visual Studio Code Now with that basic setup we are ready to create our React app: npx create-react-app name- -your-app of Depending on your development environment grap yourself a coffee, the next 2–5 minutes are not that interessting, except you like watching how all the project dependencies are downloaded and bundled together. After you are back, type the following lines into your terminal to start the app cd name- -your-app
npm start of You can now view the app in a browser by open the URL http://localhost:3000 Step 2 — Create Bitbucket Repository Now that we have our app up and running on our local machine we should create our git repository. Therefore we head over to . If you don’t have an account yet, get one its for free (important nowadays)! https://bitbucket.org When creating a new repository you need to at least give it a name and you want to set the access level to private. Most of the settings here can be changed afterwards. Well done! You’ve created your — maybe first — git repository. Time to explore the menu which you’ll now see on the lefthand side. If you want to know more about Bitbucket head over to the . official documentation Step 3 — Create Heroku App We are pretty close to deploying our app now. Head over to - again create an account if you don’t have one (IT’S FREE!). https://heroku.com On your dashboard click and choose . In the dialog: New Create new app 1. Set a - this one is unique world-wide, so it might happen that some else already picked or - be creative! App Name test123 demoapp 2. Choose the where your app should run Region Your app is now created, but before you head back to your code and git repository we need to tell our app which we’d like to use. Buildpack Go to in your AppUnder click and choose Settings Buildpacks Add buildback heroku/nodejs For deploying the app later on we also need an API Key. 1. In your Profile go to Account Settings 2. In the Section click API Key Reveal 3. Safe the key for later configuration Deploy with Bitbucket pipelines Now comes the fun part… for you actually, I learned the hard way so you don’t have to. Step 1 — Setup an express Server Go to Visual Studio Code and add a new folder called to the root of the repository. Within that folder create a file called and install express by running this command in your terminal: server server.js npm install express --save Within the file copy & paste these lines of code: server.js path = ( ); express = ( ); app = express(); publicPath = path.join(__dirname, , ); port = process.env.PORT || ;app.use(express.static(publicPath));app.get( , (req, res) => {
    res.sendFile(path.join(publicPath, ));
});app.listen(port, () => { .log( + port);
}); const require 'path' const require 'express' const const '..' 'build' const 3000 '*' 'index.html' console 'Hello World I run on PORT ' In your file change the script line for start your app to: package.json : , "start" "node server/server.js" Step 2 — Upload everything to your git repository Open a terminal windows make sure you are in the root directory of your React app and type the following lines: git init
git add .
git commit -m git remote add origin https: git push -u origin master "initial commit" //username@bitbucket.org/username/name-of-your-repository.git When you refresh the browser page you should see all the files and folders of your app we’ve just commited the the git repository. Step 3 — Setup and configure Bitbucket pipelines Now that our code is in our repository we need to enable Pipelines for our repository. Go to Repository SettingsUnder the Section click Right above Settings, go to Add a variable with name HEROKU_API_KEY and your key from aboveAdd a variable with name HEROKU_APP_NAME and the app name as value Pipelines SettingsEnable Pipelines Repository variables Now we can finally create our Pipeline configuration. In our case we want (not true, the author wants it!) to build a pipeline with following rules and steps: 1. We don’t have branches yet, so everytime someone commits something to the master branch we want to run our pipeline 2. We want to test our app first 3. We need to create a Zip file for the heroku pipline add-on 4. We want to finally push our repository to Heroku Author’s note: YES, I could push the repository to Heroku using git, but NO that was not my intention for writing this article! The file holds all the build configurations for your repository. There is really a lot you can do with this file, so if you want to dig deeper take a look at the . Here the basics of the file: bitbucket-pipelines.yml official documentation YAML : contains all your pipeline definitions. pipelines : contains the steps that run on every push if one of the following criteria is not met. default : each step starts a new Docker container with a clone of your repository, then runs the contents of your section. step script : a list of commands that are executed in sequence. script :  a pipe is more like an add-on or function, it takes some input and does stuff for you — like uploading your repository to Heroku pipe Our final file looks like this: bitbucket-pipelines.yml image: node: pipelines:
  branches:
    master:
      - step:
          name: Test App
          caches:
            - node
          script:
            - rm -rf package-lock.json
            - rm -rf node_modules
            - npm install
            - npm run test
       - step:
           name: Create artifact
           script: 
             - git archive --format=tar.gz master -o                   application.tar.gz 
         artifacts: 
           - application.tar.gz
       - step:
           name: Deploy to heroku
           deployment: production
           caches:
             - node
           script:
             - pipe: atlassian/heroku-deploy: variables:
                 HEROKU_API_KEY: $HEROKU_API_KEY
                 HEROKU_APP_NAME: $HEROKU_APP_NAME
                 ZIP_FILE: WAIT: 10.15 .3 1.1 .4 "application.tar.gz" 'true' The END Hope you liked it, even more I hope it helps you! Please let me know via comment or clap :) About Jodocus is a full-service provider for Atlassian Systems based in Hamburg, Düsseldorf and Cologne Germany. Our team combines more than 25 years of experience in building and running the solution you need. We are Atlassian experts and know how to analyze and understand your problems. Innovative and future-proof solutions are our passion and we always meet our customers at eye-level. Jodocus About the Author Jan Szczepanski specializes in project and process management and has been an Atlassian specialist since the beginning of his professional career. He was in charge of technical as well as non-technical teams in different companies and was responsible for the optimization of business processes. Today he uses his specialist knowledge, which he has acquired over many years, in to improve communication and collaboration structures.

Deploy a React App to Heroku with Bitbucket pipelines

👨‍💻_@_ 👨‍🏫_at the_ 🖥_, a_ masters level course . I’m currently a teacher assistant Técnico Lisboa , and I’m working closely with Dr. Professor Rui Cruz It Infrastructure Management and Administration @ Técnico Lisboa Why? My experience as a professor opened my mind on how to use certain tools to achieve goals that me, as a student, want to reach. As a student, I’m responsible for a student’s group, . Our project had some problems: it was very difficult and time-consuming to move from the conceptualization of an idea to its actually delivery. GCE A good, automated infrastructure was the answer to our problem. It allowed saving lots of time and effort, which translated into a better product delivered faster. This technique is meant for the ones that want to deliver as much as possible to the client as quickly as possible. How? . Feel free to check on , if you need some inspiration. One can achieve that by creating a pipeline in Heroku such: Our app runs the MEAN stack this project’s repository We have three different apps for our three different environments (development, staging, and production). All the apps have MongoLab installed, and a utility to manage logs (Papertrail). Each app is linked with a branch of our repository This allows for extreme flexibility. The workflow can be something like: Developer João dedicated to implementing a matchmaking between companies and students — branch “matchmaking”. creates a branch João codes all night. João makes a . A battery of tests is run when the request is made (we didn’t implement this so far). pull request to the branch “develop” Inês, a product manager, checks the pull request. Tests pass and she sees everything is fine. She . accepts the pull request By now, Heroku is building the development application — automatically. The whole team can see and interact with the . new version of the product, in a test environment When there is enough functionality on the branch “develop”, one (has an environment similar to production) and after that to the (production). pull request can be made to the branch “staging” branch “master” At the end of the day, you have saved lots of precious minutes, which can be very representative in one semester. Okay, I’m convinced. Let’s get to the action! The idea is to have a virtual machine as the machine which will configure our infrastructure on Heroku, through Terraform. We will manage this virtual machine using . Vagrant (up to “Requirements”, 1min read). You can achieve the same purpose without a virtual machine. In that case, skip directly to Step 2. What we will do is well condensed in the following scheme: If you want to know how Terraform works, check this tutorial Step 1: Installing Vagrant t. In this article, I show step by step how to configure i Step 2: Start the Virtual Machine Now, open your console and run: $ vagrant up --provision && vagrant ssh The virtual machine is booting, installing all the required dependencies. This may take a while. Wait a bit. Done. Nice. Kudos to you 👍 Step 3: Getting the Support Files and then clone it to your computer. Put it in the folder that is being synced with your guest machine. Fork this repository containing the support files Step 4: Analyse Terraform Files Now, replace the GitHub repositories and Heroku credentials on the file terraform-vars.tfvars. Notice what it’s being done on terraform-apps.tf. Step 5: Authenticate in Heroku through the Cli To authenticate, run: $ heroku login --interactive Put your credentials and login. Next, we need to generate an access token to Terraform to be able to communicate with Heroku. Run: $ heroku authorizations:create This will return you something like this: Go to terraform-vars.tf and paste the token obtained in the field “heroku_api_key”. Step 6: Initialize Terraform By running: $ terraform init Step 7: Deploy Your Infrastructure To know what you are going to deploy, run: $ terraform plan If everything is correct, run: $ terraform apply --auto-aprove The output should be similar to this 💯 Congratulations, your infrastructure is on! Double check it on Heroku’s dashboard: That looks fine. Don’t forget to add yourself as the owner of the pipeline I recommend you to c , by choosing a GitHub repository and associate it with a branch. For instance, we could associate the “develop” branch to the development app. When a commit occurs, it will be automatically deployed. onfigure the deployment method on each app Destroying the infrastructure With two commands you will end the experiment and shut down the virtual machine. These are: $ terraform destroy -auto-approve$ exit$ vagrant halt If you want to destroy the virtual machine you created, run: $ vagrant destroy --auto-approve 💯 Congratulations! You reached the end! 🦄 Next steps: 😀 This is a very good infrastructure for MVPs, although it might not be the best in the long run. Nextly, I’m going to talk about Kubernetes and how we can deliver an infrastructure similar to this, but much more scalable. Stay tuned Entrepreneurship 🔥 _And some tips on how to start writing._medium.com My Journey As A Computer Science Student on Medium _Imagine you are seeing the opportunity to improve yourself at every level. Would you take it?_blog.startuppulse.net Top 8 lessons I’ve learned in European Innovation Academy 2017

DevOps101: How To Provide Infrastructure To Your Startup In 6 Easy Steps

This interview was done for our Microservices for Startups ebook. Be sure to check it out for practical advice on microservices. Thanks to Nick for his time and input! is the founder and CEO of , a company that provides specialized hosted cloud services. Nick is the creator of OMC’s two hosted search as a service products, Bonsai and Websolr. Nick Zadrozny One More Cloud For context, how big is your engineering team? We are kind of a small team. We’re eight people full time and, I don’t know, three to five contractors part time at any given point depending on the projects. Are you using microservices and can you give a general overview of how you’re using them? I think I would say we started off as a monolith but that’s not entirely true. We’ve always kind of had Microservices from day one and this is day one dating back to 2009 even. The main app is mostly a user-facing dashboard. It’s an API endpoint for Heroku to provision resources and accounts and then for users to kind of get a single sign-on into their dashboard. And then if you go to Bonsai.io or websolr.com, it’s the main home page and marketing presence and create your account and log in and manage your resources and all that kind of stuff. So just the usual SaaS account and resource dashboard stuff. Websolr is managed Apache Solr as a service and Bonsai is managed ElasticSearch. We were kind of first to market with both of those. In fact, when we founded Websolr we did that by invitation of folks at Heroku as they were starting up this add-on program. Their customers wanted search engine hosting. That didn’t really exist at the time, which was really interesting. And so my original co-founder had some connections there with the folks at Heroku and some background in search. I also had background with web development and search and operations. So they kind of persuaded us: you guys should start your own business doing search hosting. It will be fun. And one thing leads to another and eight years later here we are. We have just a Rails app — that is what our customers interface with. This app is itself an API for Heroku who talks to our app to provision things for customers, and then that’s kind of the middle point for us where then we’re going to make calls out to other services to provision backend server resources and make updates to kind of middle routing layer middleware services and things like that. So that’s kind of a super high level sketch of our architecture. Did you start with a monolith and later adopt microservices? I would say we started out with Microservices from day one. We’ve always had services that would run as a kind of coordinator of server resources with Amazon independent from our main app. In the early days it was definitely much more of monolith and I think in general being a small team, there are definitely advantages to monolithic development. If so, what was the motivation to adopt microservices? How did you evaluate the tradeoffs? The microservices that we do use are done really out of a strong technical necessity. We can’t just run our Rails app on every single Solr server that we run for example. It just doesn’t make sense. And then if you were talking about that then you’re talking about all of those talking to the same central database that our app is using. So that would be a serious central point of failure for our app if we were to architect that way, because if our main database ever goes down then all of our customer instances go down. So just out of necessity we have to strongly decouple. Our Solr and our ElasticSearch clusters have to stay running no matter what’s going on with our main application. So that was the clearest separation from day one out of the technical necessity. From there, our centralized app is still kind of a monolith. We are gradually extracting very clearly defined pieces of functionality out into services. How did you approach the topic of microservices as a team/engineering organization? Was there discussion on aligning around what a microservice is? Did you change the way your team(s) were organized or operated in response to adopting microservices? I think it hasn’t been as much of an issue or question with us. We are a very engineering-heavy team. Six out of our eight full time people are engineers, kind of product and platform and operations engineers, and so I think we’re all pretty well up to speed on the concept of microservices and certainly we’re small and agile enough that we’re up to speed on modern application development deployment practices. We were a very early adopter of cloud deployments and things like Heroku’s platform model and the Twelve-Factor app model. Outsourcing to various different APIs we’ve always considered kind of a microservice approach. Now whether that’s transactional email delivery or subscription and billing management, so we’ve always made pretty decent use. I think the question for us with microservices or monolith ends up being a case by case, like we sort of evaluate that on a feature by feature basis. So when we talk about developing some new functionality, for us the question always comes up: do we build this into the Rails app or do we extract some sort of microservice that we then have the apps communicating over an API? For us it’s largely a question of the benefits of repetition. Are we going to repeat ourselves and write this code twice is essentially the question of building it into the Rails app directly, or are we going to write the bulk of the code once and write a smaller layer of integration in each app? Because each line of business for us does have its own primary monolith. They are separate apps with separate databases. So that’s really the main kind of thing that we would discuss at this point in terms of philosophically [whether] this one feature or area of functionality should be moved into a microservice or not. But I would say as far as what is a microservice and what role does it play in our architecture, I think it’s probably more clear for us as a team of mostly engineers. I would say were pretty mature in terms of our use of microservices. Have you broken a monolithic application into smaller microservices? If so, can you take us through that process? I would say we started out with Microservices from day one. How do you determine service boundaries? What was that discussion like within your team? Can you give some examples? So we have some unique requirements I think. We are a 24/7/365 company. We are an operations provider, and because of that our first responsibility is managing excellent uptime for our customers. Our customers use our service because we’re able to manage higher uptime and higher performance than they would be able to certainly at the cost of our service. So we definitely have a strong design philosophy of isolating any possible component that would be in the hot path of a customer’s traffic to their search engine. So that’s kind of rule number one in a separation boundary, is trying to clearly define when and why do we have to interact with customer resources on the backend. Most things that are going to have to either interact with customer resources directly or indirectly we’re going to want to separate from just normal kind of day-to-day product development. And so the benefits of that kind of isolation: there is the backend microservice [that] has its own API, so we can decouple the release cycles and the release management from whether we need to tweak some CSS on the front end or tweak some marketing copy or something. That can all get deployed on a way different schedule than we need to ship and update that’s going to change how we manage customer resources. That would be rolled out a lot more carefully and have its own code base that has maybe even a different kind of code review policy and things like that. I think that’s probably the first major rubric that we use for defining those service boundaries. What lessons have you learned around sizing services? [W]hen we’re looking at microservices or services internally that manage a lot of data — we store really detailed logs and analytics for all of our customer traffic. We have a proxy layer that logs all the metadata of search requests, for example. And this is like what kind of action is it, and how long did it take to complete? What was the backend server that handled it? A lot of these kinds of things so we can look at trends on operational metrics. And so this particular service was at one point responsible for collecting all the logs. So in the very early days the logs on the servers themselves grew to very unwieldy sizes or were impractical to search. And so then we start exporting all those logs into another ElasticSearch cluster so we can search within those logs. And then after a while we start hitting some definite size and scaling challenges there and you can vertically scale for a while add more capacity to that service. But in time we found it more practical to then rebuild the metrics collection one more time, and then these days we actually do a lot of pre-computation and aggregation and so now this service is not just indexing into ElasticSearch for searchability. It’s also doing a lot of aggregating and we store metrics in Cassandra. So the flexibility to kind of recompose how things are persisted. Starting with on disk log files to then Elk Stack, to then Elk plus Cassandra was one really nice benefit of having some of that metadata just shipped directly off the server and onto some other service within our backend. And we are able to do all of that — and again this is is directly adjacent to the customer hot path — we are able to do manage releases in a way that was effective that led us to kind of do some major capability and capacity improvements without risking any kind of impact to actual customer traffic. How have Microservices impacted your development process? Your ops and deployment processes? What were some challenges that came up and how did you solve them? Can you give some examples? Deployment has definitely sped up every time we’ve introduced an effective microservice. We’ve always seen an increase in performance because we have a lot more confidence when we clearly define these boundaries that when you’re making a change on one side or another you understand the scope of what’s being impacted when you’re reviewing code and planning for deployment. So we now have a deployment process where a lot of times if we’re going to add a new functionality that depends on a microservice we can dark deploy the functionality on the backend first and get that working correctly. And then only later once once it’s done and working well we can shift the interface for it in the front end. So decoupling those two means you’re able to make smaller changes more frequently and that just really increases our confidence in the changes that we’re making. It improves our ability to conceptualize what has been changed and and not have to worry about code that’s all entangled and creating sort of unnecessary and unclear points where your software is coupled to itself. That’s just incredibly valuable in operations as well. If you aren’t constantly vigilant in maintaining an ability to deploy frequently and deploy very small changes and to validate them, then you kind of get stuck in this pattern where you’re releasing once a quarter or God forbid, once a year. And it’s just like pulling teeth to get all the resources to make it happen and then by the time it’s actually done, now you’re planning for your next one. Have microservices impacted the way you approach testing? What are lessons learned or advice around this? Can you give some examples? Yeah, it has. And we’re still working through some stuff on this actually. I find it pretty interesting. I mean it is nice from testing perspective. You’re able to have much more focused unit tests on the kind of again you have the service boundary, right? And so one side of the service boundary you have the microservice; on the other side of the boundary you have the client that is consuming it. And so you’re able to write tests with both in mind. The unit tests that are testing the microservice itself — it’s a little bit more of an implementation detail. You have a little more freedom and flexibility as the owner of a microservice to kind of change around your implementation of things and that goes with changing your tests. And so by clearly defining the boundary, like maybe there does not need to be a change to the API itself. I think you just promote a lot more productivity. I think tests are more clear, more productive, they’re a little more disposable. It’s a tool that serves a purpose for a time in the context that it needs to exist in. But from the perspective of the app that’s consuming the microservice it really doesn’t care about that stuff. I find that that’s really a really clear benefit when it comes to testing. We have a lot fewer tests in our main primary monolith that test the behavior of the microservice because we’re able to trust at a higher level that it is serving the job, that it is doing its job that it needs to do. And so that keeps the monolith a lot more focused. It prevents coupling back to what may be implementation details in the microservice. And then it just allows a lot more flexibility within the microservice itself to kind of change and adapt. It’s kind of more of a development practice thing than specifically testing. But yeah, I think that’s kind of the main thing we get is just the flexibility to use the right tool for the job and to focus on the code base that you’re working in directly, focusing on what it is responsible for. So changing the way we approach testing, we would then — I think in our microservices, most of them bundle a client as part of the project. We may have a Java drop wizard microservice in the back end and it would have a Ruby gem client bundled in that repo that would then be consumed by the Rails app. And so we would actually drive the integration test and the microservice itself using its own client. That’s one of the nice things about that kind of testing. We would still have Java unit tests as well. But ultimately the Ruby client is where we would kind of put all the integration tests and put examples of how this thing expects to be used. And if a contract was ever violated on the API level then we have a place to go where we would then create a regression test. Have you run into issues managing data consistency in a microservice architecture? If so, describe those issues and how you went about solving them. We help our customers with this question all the time. I mean for us data consistency is not as much of a problem because we do have that monolith up front and that, because it’s talking to PostgreSQL, basically PostgreSQL transactions give us all of the consistency we need. The way our models are defined mostly as an event based design in our Rails app gives us pretty much everything we need in terms of resolving any potential data race or concurrency or conflict situation. So our challenges on that stuff are less than what our customers might have. One of our microservices does have a distributed data store that’s built into it. But fortunately it is kind of a last right wins design. There are three layers of this for me. Our business is the management of distributed data services, namely ElasticSearch is a big one and it was designed from day one to be distributed. So over the years just being engineers staying up with cloud architecture, we’ve had to remain really educated on the distributed systems engineering subjects. So our main customer base and accounts and cluster management and resource management, that’s not necessarily a distributed problem because it goes through a central database. But when it comes to coordinating some of our data services, that can become a distributed data problem and so this is where we have at times had to rely on databases that are specifically built for that environment. So we do make use of a Zookeeper, which is designed for consistency in a distributed environment. We also have a different data service that is designed for high availability, so it may not always be 100 percent consistent, but it is always 100 percent available. And so when we design microservices in the backend there’s always going to be a question of what is the fundamental backing data storage and what are the tradeoffs with respect to data consistency and availability, and how does that impact our decisions? So we have all three. We have central, we have decentralized but optimized for consistency, and we have decentralized but optimized for availability. That said, managing consistency is really interesting when you look at say one of our customers integrating with these search engine service like ours because now they have to make changes in their primary data store and those changes have to get out to their search engine and be searchable. And sometimes they need to make updates to resources and coordinate the consistency of those updates, especially if there are many updates happening in quick succession to a single document for example. So we do a lot of education with our customers through our support channels. We have half a dozen different kinds of best practices that we can recommend to people when it comes to managing data consistency. I think really the best thing for an engineer in any sort of microservices world, in a distributed world, I think the best thing to consider is don’t create data consistency problems if you don’t have to. Just use PostgreSQL or some other reliable central data store. That’s not always the right answer because that becomes a single point of failure. I think the bottom line is engineers really owe it to themselves to become educated on the subject of data consistency and designing data persistence and the tradeoffs of consistency and availability and just pure centralization. Thanks again to Nick for his time and input! This interview was done for our Microservices for Startups ebook. Be sure to check it out for practical advice on microservices. . This article was originally published at buttercms.com . It’s part of an interview series for the book Microservices for Startups Sign up to get new chapters emailed to you as soon as they’re released. If you’ve enjoyed this article, please help it spread by clapping below! For more content like this, follow us on Twitter and subscribe to our blog. And if you want to add a blog or CMS to your website without messing around with Wordpress, you should try Butter CMS .

API HERO

apihero.run

1-10 employees

Since 2022

COMPANY RANKING

API HERO

EVERGREEN INDEX #1299

Representing this company? Click here to claim and customize this page!

API Hero WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!

API HERO

apihero.run

1-10 employees

Since 2022

COMPANY RANKING

API HERO

EVERGREEN INDEX #1299

RELATED COMPANIESRANK

STORIES ABOUT API HERO

Representing this company? Click here to claim and customize this page!

API Hero WIKI

Apply to get your Tech Company News Page live on HackerNoon.com today!