The 7 Biggest Lessons I’ve Learned By Building A Twitter Bot 🐤🤖

Today, I am going to share with you the most important lessons I have learned by building Release Butler — a that tweets beautifully styled changelogs of popular frontend frameworks and libraries like Angular, React, Vue or Webpack… Twitter bot Example of a tweet made by Release Butler Release Butler got very positive welcome from the community with lots of engagement and more than 600 followers during the first month after it’s release in the beginning of April 2018. Release Butler got pretty good traction during the first month after its release! The lessons (tldr) ⌛ Don’t try to save time by skipping Typescript 🏋 Async /await is great until it isn’t & exception handling can be tough 🔑 Logging is the key 🎉 Going on the frontend without a framework can be refreshing vanilla 🤗 Embrace the CLI and build your own tools 🏛️ is a great platform for building bots — node vs Docker “Now” 🥊 Not all APIs were created equal — GitHub vs Twitter 🛡️ : Basic security BONUS The context I am currently for the ninth month. Believe it or not adventure can become repetitive too… You can easily find yourself yearning for “something” to do… traveling around the world I am very happy to be part of the wonderful tech industry and love the idea of giving back to the community by trying to create something useful. Lately, I have been very inspired by amazing project which enables you to create and share beautiful code snippets. carbon.now.sh Release Butler lives at releasebutler.now.sh This led to an idea to create similar service, but for the changelogs instead. The concept later cristalyzed in what is now called Release Butler. Besides being a , it comes also with a website which enables you to download changelog of any GitHub hosted library which uses GitHub releases or Changelog.md file. Twitter bot , a twitter bot that helps you to stay up to date with releases of popular frontend libraries… Follow Release Butler ⌛ 1. Don’t try to save time by skipping Typescript The origins of Release Butler can be described more like a disorganized experimentation than a well-thought project with detailed architecture, milestones and what-not… In the beginning, a single small moment of laziness resulted in skipping of Typescript which has now major consequences… Let me just tell you, use Typescript, it’s worth it! The effort to start using Typescript in a node project is really low. All we need to do is run and then execute our application using instead . We don’t even have to type every variable or function from the start. Types can be added gradually, every time one of the interfaces becomes more stable. npm i -S typescript ts-node ts-node . node . While some of the pain can be prevented by a more detailed architecture docs before the start of the project, it is NOT reasonable to expect that project will be designed perfectly from the beginning. Requirements tend to evolve and we often need to reorganize our code base. Code reuse opportunities and abstractions become more apparent as we add more services and components. Refactoring can become a major pain even in a small project like Release Butler. Every time we have to change interface of any of our services without a strong type system we are at risk of breaking some distant part of our application But what about the tests? This wouldn’t be a concern if we had a proper test coverage, right? The main disadvantage of tests to solve this kind of problems is that we have to write them in the first place… I am a huge fan of writing unit and integration tests for the business logic On the other hand, seeing lots code that checks for the presence or of a function arguments and corresponding unit tests is a big code smell… 💩 typeof Typescript can dramatically reduce amount of code needed to guarantee basic correctness of the application. Besides that, most of the popular editors come with substantial refactoring capabilities when used together with a typed language like Typescript. 🏋 2. Async / await is great until it isn’t & exception handling can be tough Async / await is a newish addition to the JavaScript language. I haven’t really used it before and worked directly with the promises instead… Simplified example of using async / await in Release Butler codebase The main premise of async / await is to make async code look just like plain old sync stuff. Every line of code executes in the top to bottom fashion while waiting (await) for the result of asynchronous operation when necessary. It’s also great for more complex orchestration of multiple async services resulting in much “flatter” code compared to callbacks or promises. The biggest challenge of using async / await comes with the implementation of exception handling Attaching handler at the end of the local promise chain is much more readable than wrapping different and parts of the execution in blocks. .catch(err => { /* handle error /*}) often nested try / catch As the project grows, the number of async functions becomes larger and orchestration more complex. It is no longer enough to just wrap every single async execution with a block and handle all exceptions locally. try / catch A lot of consideration has to go into figuring out how to handle unexpected behavior with respect to the overall desirable functionality. We tend to end up with following cases: exception should be handled at the current level and result in valid state exception should be handled partially at the current level (for example we log error) and re-thrown for handling up in the execution stack exception should be not handled at all at the current level and will be caught by some of the parent functions This can get tricky. Async functionality offered by one of the services can be consumed by different parts of application with wildly different expectations about what will happen when things go wrong… 🔑 3. Logging is the key In frontend development, we’re used to getting immediate visual feedback for every change of code we just authored. On the other hand, long running backend process doesn’t provide any useful information out of the box. By default, we know only if it runs, have stopped or crashed… Logging is THE solution for getting useful insight into current state of our backend system and the easiest thing to do is to start logging everything Incomming API request? Database query or script execution? No problem, logs got us covered… But as with everything, it is important to strike a nice balance — too much and we will get drowned in the noise, too little and we’re running blind in the darkness of the night… Iteration on what and how to log is a ongoing process… Current logs for a single Release Butler execution provide quick overview of the released projects and their versions Current logs are very concise. A lot of information was removed over the course of iteration… timestamp — provides its own timestamp now platform — as it turned out, it is much more interesting to know what has happened instead of what didn’t happen No new version for project: names of tracked projects — five tracked projects can fit into one line but the number soon increased to more than 20, that’s a lot of useless repetion list of all released versions — it is nice to know what was released since last deployment but it can get problematic if bot runs for months and the list grows to hundreds of versions, the compromise was to log number of releases and the last version various implementation related info — details of every request or database query can be useful but adds too much noise during normal operation so it is better to use log level which has to be enabled manually when needed debug 🎉 4. Going on the frontend without ANY framework can be refreshing vanilla Inspiration for Release Butler came partially from a great project called . It’s website which enables creation of beautifully looking code snippets. Release Butler strives to enable you to instead. carbon.now.sh get and share beautifully looking changelogs Release Butler is a twitter bot but comes also with which enables you to get beautifully styled changelog of any GitHub hosted library which uses GitHub releases or Changelog.md file website The page itself can be described as a marketing website with a single form for retrieving changelogs. For a use case like this, using a full-blown framework would be an unnecessary overkill. I am a huge fan of Angular framework but it is very important to be able to set aside personal preferences and choose a right tool for the job Currently, there is only a single 5kB . No npm, no build scripts, no minification, no jQuery just couple of lines of honest vanilla Javascript. Refreshing 🍹! app.js file 🤗 5. Embrace the CLI and build your own tools At its core, a is just a fancy name for a script which repeatedly executes functionality in a loop or in reaction to some events. bot For example, Release Butler checks if there was a new release of one of the tracked GitHub projects every ten minutes. Unfortunately, the time based nature of the execution is not very practical for trying things out during the development. We can of course reduce execution delay but that won’t help too much. Imagine that we still have to wait at least 30 seconds to see the result of a single styling change. What we really need is a way to to execute functionality on demand with as little friction as possible… Similarly, interacting with many 3rd party APIs and environments brings issues on its own. It would take prohibitively huge amount of time to mock all these environments just for the development purposes. It is easy to find yourself developing against the and that’s OK 😉 “prod” Early development and a lot of experimentation means that you will need to revert data and other side-effects on a hourly basis which might not be practical using the tools provided by these platforms. Deleting tweets manually can be tough… Now go, delete the rest of the 50 generated test tweets 😬 As a web developers, we’re naturally inclined to think about creating specialized admin app which exposes functionality of the bot with a slick UI. This is a very common productivity trap which could easily consume crazy amounts of time. We’re much better off focusing on the development of the core functionality. There must be a better way… Embrace the CLI to expose small chunks of functionality Adding command line interface to our app requires only minor effort and delivers massive results! There are many different libraries that can help us building CLI and one of my favorite is … yargs Yargs helps you build interactive command line tools, by parsing arguments and generating an elegant user interface. Example of a custom CLI tooling to help with development of Release Butler by removing data from 3rd party environments (“node .” stands for running current folder as a node application, it will look into package.json and execute file referenced by the “main” property) Implementation is straight forward. Just use yargs DSL to specify supported commands and parameters… Yargs uses nice expressive DSL to define supported commands and configuration flags 🏛️ 6.️ “Now” is a great platform for building bots In 2018 we’re used to deploying our code directly into the cloud with a few keystrokes. No time-consuming server provisioning or configuration is necessary. There are plenty of platform as a service providers (PaaS) with a that helps us build our prototypes without any financial costs. free tier Unfortunately, the problem with many offerings is that there is usually a limit on the uptime of the free instances. This is problematic for a bot development. Bots are supposed to be up and running all the time. “free tier” I have been searching for a solution and discovered that a rather newish PaaS company called with a very user friendly platform called offers three permanently running instances for free! Zeit now — I am NOT related to Zeit in way, I just enjoyed working with now Disclaimer Now supports deployment for static websites, node apps and docker containers. Node deployment is a natural choice to start with and it works very well for most standard use cases. Release Butler needs to be able to take a customized screenshot of a project changelog on GitHub. This is done with the help of which comes with the bundled when installed from npm… puppeteer headless chrome As it turned out puppeteer works really great on the development machine but breaks down when deployed to Zeit because of missing OS level dependencies The solution was to switch to the docker deployment I have never used docker before so I was a little bit worried about how much do I have to learn before getting things up and running. After a bit of googling I was able to put together Dockerfile which was based on node and installed all the missing dependencies during the deployment process. The biggest downside of using docker is that the deployment process became . Installing all the missing OS dependencies during the every single deployment just takes too much time. painfully long Luckily it is possible to pre-build our own docker images and store them on Docker Hub (similar to GitHub). This dramatically reduces the deployment times almost to the levels of plain node deployment. Docker is a huge topic in itself but it seems to be worth it to learn at least some basics, it can save your day! 🥊 7. Not all APIs were created equal — GitHub vs Twitter Building of a bot rarely happens in isolation. Bot is usually used as a replacement for real user interaction on the target platform. Platforms often provide application programming interface (API) to expose some of their functionality to the developers. Release Butler is currently interacting with two platforms — Twitter and GitHub. GitHub API Release Butler needs a way to determine if there was a new relese of a tracked library. It needs to retrieve latest released versions from the GitHub repository and compare them to the locally stored versions from its database. Consuming GitHub API is straight forward. Readonly endpoints can be consumed anonymously but there is a to the amount of requests that can be made per hour from single IP address. limit We can fix this by generating personal access token which will be sent together with every request. Every GitHub user can generate personal access tokens by navigating to and using generate token functionality. settings > developer settings > personal access tokens Requests themselves can be performed using which is available natively in every modern browser but in node we have to install it using . It has a simple API and Promise based interface which plays nicely with async / await syntax. fetch npm i -S node-fetch Just one TOKEN is all we need to start using GitHub API Twitter API Being a Twitter bot, Release Butler needs to be able to tweet about new releases of tracked libraries. My expectation about the developer experience of using Twitter API proved to be very very wrong Complications started immediately after creating new Twitter account for Release Butler. There is no simple “get a token” and you are ready to go solutions… Instead, what we have to do is associated with the account. Oops, this doesn’t work if you didn’t provide and verify real phone number first… Seriously?! create a Twitter app The documentation, while very extensive, proved to be pretty confusing… There simple step by step guide to how to successfully make a request to the Twitter API… IS NO Documentation mentions some ready made twitter client libraries on NPM, but unfortunately they are quite dated, callback based and do don’t support every endpoint out of the box… Digging around uncovered it’s all about OAuth. Speed-learning OAuth and a naive try to re-implement it ended up in failure so it was back to npm hunt for suitable package. Luckily this was a more fruitful endeavor and works just fine.. npm i -S oauth Simplified example of code needed to consume Twitter API Simple but complete “make request example” can save us many hours of digging around the docs, Twitter please! The library is callback based so it is wrapped and promisified in the to play nicely with async / await syntax. ouath real Release Butler implementation 🛡️ BONUS: Basic security I hope it goes without saying that we should never ever hard-code and commit our tokens and secrets into version control system like git. Application should retrieve all necessary configuration from its environment. Now supports file where we can specify both public and secret environment variables. Secrets are not specified directly but using a placeholder. now.json For local development, we can define placeholder values in the which we also add into so we can be sure we will never commit this file into our repository. now-secrets.json .gitignore For prod environment, we can define placeholder values using (check out for more details). now secret add official docs Example of a now.json with public and secret environment variables. Secrets contain placeholders starting with ‘@’ character. That’s it for today! I hope can help you to stay up to date with frontend releases and that you will ! Please support this article with your 👏👏👏 to help it spread to a wider audience 🙏. Release Butler give it a try by following it on Twitter And never forget, future is bright Obviously the bright future (📷 by ) Mohamed Thasneem If you made it this far and feel like you want to learn something more about interesting frontend topics, feel free to give a try to some of the following articles 😉 _Yeah, the title, I know… but I had to try it at least once in my life 😬😂_medium.com Medium Hates Him! See How He Improved Their Stats Page With This One Simple Trick _Managing multiple requests with RxJS can be done in multiple ways. Each with its pros & cons. Learn when to use…_blog.angularindepth.com Practical RxJS In The Wild 🦁— Requests with concatMap() vs mergeMap() vs forkJoin() 🥊 _While very controversial, the new npm release 5.7.0 brings some amazing features which will have noticeable positive…_medium.com How To Speed Up Continuous Integration Build With New NPM CI And package-lock.json _Introducing Release Butler — A Twitter Bot That Helps You To Stay Up To Date With Releases Of Popular Frontend…_medium.com How To Stay Up To Date With Releases Of Popular Frameworks