The 7 Biggest Lessons I’ve Learned By Building A Twitter Bot 🐤🤖

Today, I am going to share with you the most important lessons I have learned by building Release Butler — a Twitter bot that tweets beautifully styled changelogs of popular frontend frameworks and libraries like Angular, React, Vue or Webpack…

Example of a tweet made by Release Butler

Release Butler got very positive welcome from the community with lots of engagement and more than 600 followers during the first month after it’s release in the beginning of April 2018.

Release Butler got pretty good traction during the first month after its release!

The lessons (tldr)

1. ⌛ Don’t try to save time by skipping Typescript
2. 🏋 Async /await is great until it isn’t & exception handling can be tough
3. 🔑 Logging is the key
4. 🎉 Going vanilla on the frontend without a framework can be refreshing
5. 🤗 Embrace the CLI and build your own tools
6. 🏛️ “Now” is a great platform for building bots — node vs Docker
7. 🥊 Not all APIs were created equal — GitHub vs Twitter

🛡️ BONUS: Basic security

The context

I am currently traveling around the world for the ninth month. Believe it or not adventure can become repetitive too… You can easily find yourself yearning for “something” to do…

I am very happy to be part of the wonderful tech industry and love the idea of giving back to the community by trying to create something useful. Lately, I have been very inspired by amazing carbon.now.sh project which enables you to create and share beautiful code snippets.

Release Butler lives at releasebutler.now.sh

This led to an idea to create similar service, but for the changelogs instead. The concept later cristalyzed in what is now called Release Butler.

Besides being a Twitter bot, it comes also with a website which enables you to download changelog of any GitHub hosted library which uses GitHub releases or Changelog.md file.

Follow Release Butler, a twitter bot that helps you to stay up to date with releases of popular frontend libraries…

⌛ 1. Don’t try to save time by skipping Typescript

The origins of Release Butler can be described more like a disorganized experimentation than a well-thought project with detailed architecture, milestones and what-not…

In the beginning, a single small moment of laziness resulted in skipping of Typescript which has now major consequences…

Let me just tell you, use Typescript, it’s worth it!

The effort to start using Typescript in a node project is really low. All we need to do is run npm i -S typescript ts-node and then execute our application using ts-node . instead node . . We don’t even have to type every variable or function from the start. Types can be added gradually, every time one of the interfaces becomes more stable.

While some of the pain can be prevented by a more detailed architecture docs before the start of the project, it is NOT reasonable to expect that project will be designed perfectly from the beginning.

Requirements tend to evolve and we often need to reorganize our code base. Code reuse opportunities and abstractions become more apparent as we add more services and components. Refactoring can become a major pain even in a small project like Release Butler.

Every time we have to change interface of any of our services without a strong type system we are at risk of breaking some distant part of our application

But what about the tests? This wouldn’t be a concern if we had a proper test coverage, right?

The main disadvantage of tests to solve this kind of problems is that we have to write them in the first place…

I am a huge fan of writing unit and integration tests for the business logic

On the other hand, seeing lots code that checks for the presence or typeof of a function arguments and corresponding unit tests is a big code smell… 💩

Typescript can dramatically reduce amount of code needed to guarantee basic correctness of the application. Besides that, most of the popular editors come with substantial refactoring capabilities when used together with a typed language like Typescript.

🏋 2. Async / await is great until it isn’t & exception handling can be tough

Async / await is a newish addition to the JavaScript language. I haven’t really used it before and worked directly with the promises instead…

Simplified example of using async / await in Release Butler codebase

The main premise of async / await is to make async code look just like plain old sync stuff. Every line of code executes in the top to bottom fashion while waiting (await) for the result of asynchronous operation when necessary.

It’s also great for more complex orchestration of multiple async services resulting in much “flatter” code compared to callbacks or promises.

The biggest challenge of using async / await comes with the implementation of exception handling

Attaching .catch(err => { /* handle error /*}) handler at the end of the local promise chain is much more readable than wrapping different and often nested parts of the execution in try / catch blocks.

As the project grows, the number of async functions becomes larger and orchestration more complex. It is no longer enough to just wrap every single async execution with a try / catch block and handle all exceptions locally.

A lot of consideration has to go into figuring out how to handle unexpected behavior with respect to the overall desirable functionality. We tend to end up with following cases:

1. exception should be handled at the current level and result in valid state
2. exception should be handled partially at the current level (for example we log error) and re-thrown for handling up in the execution stack
3. exception should be not handled at all at the current level and will be caught by some of the parent functions

This can get tricky. Async functionality offered by one of the services can be consumed by different parts of application with wildly different expectations about what will happen when things go wrong…

🔑 3. Logging is the key

In frontend development, we’re used to getting immediate visual feedback for every change of code we just authored. On the other hand, long running backend process doesn’t provide any useful information out of the box. By default, we know only if it runs, have stopped or crashed…

Logging is THE solution for getting useful insight into current state of our backend system and the easiest thing to do is to start logging everything

Incomming API request? Database query or script execution? No problem, logs got us covered…

But as with everything, it is important to strike a nice balance — too much and we will get drowned in the noise, too little and we’re running blind in the darkness of the night…

Iteration on what and how to log is a ongoing process…

Current logs for a single Release Butler execution provide quick overview of the released projects and their versions

Current logs are very concise. A lot of information was removed over the course of iteration…

• timestamp — now platform provides its own timestamp
• No new version for project: <project-name> — as it turned out, it is much more interesting to know what has happened instead of what didn’t happen
• names of tracked projects — five tracked projects can fit into one line but the number soon increased to more than 20, that’s a lot of useless repetion
• list of all released versions — it is nice to know what was released since last deployment but it can get problematic if bot runs for months and the list grows to hundreds of versions, the compromise was to log number of releases and the last version
• various implementation related info — details of every request or database query can be useful but adds too much noise during normal operation so it is better to use debug log level which has to be enabled manually when needed

🎉 4. Going vanilla on the frontend without ANY framework can be refreshing

Inspiration for Release Butler came partially from a great project called carbon.now.sh. It’s website which enables creation of beautifully looking code snippets. Release Butler strives to enable you to get and share beautifully looking changelogs instead.

Release Butler is a twitter bot but comes also with website which enables you to get beautifully styled changelog of any GitHub hosted library which uses GitHub releases or Changelog.md file

The page itself can be described as a marketing website with a single form for retrieving changelogs. For a use case like this, using a full-blown framework would be an unnecessary overkill.

I am a huge fan of Angular framework but it is very important to be able to set aside personal preferences and choose a right tool for the job

Currently, there is only a single 5kB app.js file. No npm, no build scripts, no minification, no jQuery just couple of lines of honest vanilla Javascript. Refreshing 🍹!

🤗 5. Embrace the CLI and build your own tools

At its core, a bot is just a fancy name for a script which repeatedly executes functionality in a loop or in reaction to some events.

For example, Release Butler checks if there was a new release of one of the tracked GitHub projects every ten minutes.

Unfortunately, the time based nature of the execution is not very practical for trying things out during the development. We can of course reduce execution delay but that won’t help too much. Imagine that we still have to wait at least 30 seconds to see the result of a single styling change.

What we really need is a way to to execute functionality on demand with as little friction as possible…

Similarly, interacting with many 3rd party APIs and environments brings issues on its own. It would take prohibitively huge amount of time to mock all these environments just for the development purposes. It is easy to find yourself developing against the “prod” and that’s OK 😉

Early development and a lot of experimentation means that you will need to revert data and other side-effects on a hourly basis which might not be practical using the tools provided by these platforms.

Deleting tweets manually can be tough…

Now go, delete the rest of the 50 generated test tweets 😬

As a web developers, we’re naturally inclined to think about creating specialized admin app which exposes functionality of the bot with a slick UI.

This is a very common productivity trap which could easily consume crazy amounts of time. We’re much better off focusing on the development of the core functionality. There must be a better way…

Embrace the CLI to expose small chunks of functionality

Adding command line interface to our app requires only minor effort and delivers massive results!

There are many different libraries that can help us building CLI and one of my favorite is yargs…

Yargs helps you build interactive command line tools, by parsing arguments and generating an elegant user interface.

Example of a custom CLI tooling to help with development of Release Butler by removing data from 3rd party environments (“node .” stands for running current folder as a node application, it will look into package.json and execute file referenced by the “main” property)

Implementation is straight forward. Just use yargs DSL to specify supported commands and parameters…

Yargs uses nice expressive DSL to define supported commands and configuration flags

🏛️ 6.️ “Now” is a great platform for building bots

In 2018 we’re used to deploying our code directly into the cloud with a few keystrokes. No time-consuming server provisioning or configuration is necessary. There are plenty of platform as a service providers (PaaS) with a free tier that helps us build our prototypes without any financial costs.

Unfortunately, the problem with many “free tier” offerings is that there is usually a limit on the uptime of the free instances. This is problematic for a bot development. Bots are supposed to be up and running all the time.

I have been searching for a solution and discovered that a rather newish PaaS company called Zeit with a very user friendly platform called now offers three permanently running instances for free!

Disclaimer — I am NOT related to Zeit in way, I just enjoyed working with now

Now supports deployment for static websites, node apps and docker containers. Node deployment is a natural choice to start with and it works very well for most standard use cases.

Release Butler needs to be able to take a customized screenshot of a project changelog on GitHub. This is done with the help of puppeteer which comes with the bundled headless chrome when installed from npm…

As it turned out puppeteer works really great on the development machine but breaks down when deployed to Zeit because of missing OS level dependencies

The solution was to switch to the docker deployment

I have never used docker before so I was a little bit worried about how much do I have to learn before getting things up and running. After a bit of googling I was able to put together Dockerfile which was based on node and installed all the missing dependencies during the deployment process.

The biggest downside of using docker is that the deployment process became painfully long. Installing all the missing OS dependencies during the every single deployment just takes too much time.

Luckily it is possible to pre-build our own docker images and store them on Docker Hub (similar to GitHub). This dramatically reduces the deployment times almost to the levels of plain node deployment.

Docker is a huge topic in itself but it seems to be worth it to learn at least some basics, it can save your day!

🥊 7. Not all APIs were created equal — GitHub vs Twitter

Building of a bot rarely happens in isolation. Bot is usually used as a replacement for real user interaction on the target platform. Platforms often provide application programming interface (API) to expose some of their functionality to the developers.

Release Butler is currently interacting with two platforms — Twitter and GitHub.

GitHub API

Release Butler needs a way to determine if there was a new relese of a tracked library. It needs to retrieve latest released versions from the GitHub repository and compare them to the locally stored versions from its database.

Consuming GitHub API is straight forward. Readonly endpoints can be consumed anonymously but there is a limit to the amount of requests that can be made per hour from single IP address.

We can fix this by generating personal access token which will be sent together with every request. Every GitHub user can generate personal access tokens by navigating to settings > developer settings > personal access tokens and using generate token functionality.

Requests themselves can be performed using fetch which is available natively in every modern browser but in node we have to install it using npm i -S node-fetch . It has a simple API and Promise based interface which plays nicely with async / await syntax.

Just one TOKEN is all we need to start using GitHub API

Twitter API

Being a Twitter bot, Release Butler needs to be able to tweet about new releases of tracked libraries.

My expectation about the developer experience of using Twitter API proved to be very very wrong

Complications started immediately after creating new Twitter account for Release Butler.

There is no simple “get a token” and you are ready to go solutions…

Instead, what we have to do is create a Twitter app associated with the account. Oops, this doesn’t work if you didn’t provide and verify real phone number first… Seriously?!

The documentation, while very extensive, proved to be pretty confusing… There IS NO simple step by step guide to how to successfully make a request to the Twitter API…

Documentation mentions some ready made twitter client libraries on NPM, but unfortunately they are quite dated, callback based and do don’t support every endpoint out of the box…

Digging around uncovered it’s all about OAuth. Speed-learning OAuth and a naive try to re-implement it ended up in failure so it was back to npm hunt for suitable package. Luckily this was a more fruitful endeavor and npm i -S oauth works just fine..

Simplified example of code needed to consume Twitter API

Simple but complete “make request example” can save us many hours of digging around the docs, Twitter please!

The ouath library is callback based so it is wrapped and promisified in the real Release Butler implementation to play nicely with async / await syntax.

🛡️ BONUS: Basic security

I hope it goes without saying that we should never ever hard-code and commit our tokens and secrets into version control system like git. Application should retrieve all necessary configuration from its environment.

Now supports now.json file where we can specify both public and secret environment variables. Secrets are not specified directly but using a placeholder.

For local development, we can define placeholder values in the now-secrets.json which we also add into .gitignore so we can be sure we will never commit this file into our repository.

For prod environment, we can define placeholder values using now secret add <key> <value> (check out official docs for more details).

Example of a now.json with public and secret environment variables. Secrets contain placeholders starting with ‘@’ character.

That’s it for today!

I hope Release Butler can help you to stay up to date with frontend releases and that you will give it a try by following it on Twitter! Please support this article with your 👏👏👏 to help it spread to a wider audience 🙏.

And never forget, future is bright

Obviously the bright future (📷 by Mohamed Thasneem)

If you made it this far and feel like you want to learn something more about interesting frontend topics, feel free to give a try to some of the following articles 😉

Medium Hates Him! See How He Improved Their Stats Page With This One Simple Trick_Yeah, the title, I know… but I had to try it at least once in my life 😬😂_medium.com

Practical RxJS In The Wild 🦁— Requests with concatMap() vs mergeMap() vs forkJoin() 🥊_Managing multiple requests with RxJS can be done in multiple ways. Each with its pros & cons. Learn when to use…_blog.angularindepth.com

How To Speed Up Continuous Integration Build With New NPM CI And package-lock.json_While very controversial, the new npm release 5.7.0 brings some amazing features which will have noticeable positive…_medium.com

How To Stay Up To Date With Releases Of Popular Frameworks_Introducing Release Butler — A Twitter Bot That Helps You To Stay Up To Date With Releases Of Popular Frontend…_medium.com