The Winds 2.0 Stack

Winds started out as a simple example app for Stream, but thanks to an outpouring of support from our awesome community we decided to focus more time and energy on the project. The initial response around Winds 2.0 has exceeded all of our expectations. Since launching in mid-May the application ranked #1 on Hacker News for over a day, has 5,500 stars (and counting) on Github and became a trending app on Product Hunt. Going into it, there was no way to anticipate how popular Winds 2.0 would become — would it be an absolute hit or an epic failure and waste of time?? The team enjoys building example apps so I knew it wouldn’t be a total loss, and it’s been rewarding to see this new iteration being used so extensively in the first month after release. The tech stack for Winds is completely different from that of Stream. You may have seen the blogpost that StackShare how Stream powers the activity feeds for 300 million users using Go, RocksDB and Raft. Winds, however, is based on , , , , , and . wrote about Node.js MongoDB Atlas Express PM2 Bull Babel React To get started with Winds 2.0, you try the web version or download the application , or if you feel more adventurous head over to and spin it up locally. Next up, let’s talk a little bit about the Winds 2.0 stack and why we chose to go with the technologies we did (and why we chose to build Winds in the first place!). here GitHub RSS is a Broken Experience 😭 We realize that many RSS power users are developers, designers and journalists. One of our goals with Winds is to answer the questions we had been asking ourselves: What if a community of developers and designers could create an RSS experience that’s simplistic and polished? Could this reverse the downward spiral of less users taking advantage of the technology and more publications dropping support? The future of RSS is uncertain at best. Our hope with this project is to make a contribution to #ReviveRSS. Why JavaScript/Node…? 🤔 Another core goal for Winds is to enable a wide range of developers to contribute. We want it to be easy for to be able to notice something they don’t like about their RSS/Podcast experience and easily submit a pull request with a fix or addition. anyone If you were brave enough to explore the codebase, you probably noticed that we’re using JavaScript for everything — both front and backend. Most of our team is experienced with Go and Python, so Node was not an obvious choice for this example app. What’s funny about JavaScript is how many people complain about it being an inadequate language. Sure, it has its quirks — single threaded, callback hell, etc.– but we believe that it’s possible to build great software in any language. For Winds, JavaScript has been a great choice to foster a community around the project. More importantly, JavaScript’s maturity has started to shine with the added support of the syntax. Async/Await Sure… there will be haters who refuse to acknowledge that there is anything remotely positive about JavaScript (there are even rants on about Node.js.); however, without writing completely in JavaScript, would not have seen the results we did. Here’s a quick breakdown of some of the reasons why chose JavaScript: Hacker News we we Nearly every developer knows or can, at the very least, read JavaScript With and Node.js v10.x.x, it’s become a very capable language ES6 / is powerful and easy to use ( ) Async Await Async/Await vs Promises allows us to experiment with next-generation JavaScript (features that are not in the official JavaScript spec yet) Babel allows us to consistently install packages quickly (and is filled with ) Yarn tons of new tricks DevOps 🖥️ It’s rare that you hear about how a company deploys and manages code. Being that Winds 2.0 is open-source, we wanted to share a few of the tools we use to get the job done when it comes to getting our code from our machines up to the server. The web version of Winds is statically hosted on S3 with CloudFront. In all, it costs us a few dollars a month to host. Every desktop version of the application is deployed inside of Electron, allowing us to bridge the gap between web and desktop. As for the back-end API, that’s another story. We have a rather decent deploy flow going on to ensure stability and maintainability. Here’s the rundown: All code is stored on GitHub We manually kick off builds on AWS using a combination of and Fabric Boto create a fresh Winds environment consisting of instances, (ASG), (ELB), and a instance CloudFormation EC2 Auto Scaling Groups Application Load Balancer Redis AWS CCM stores and retrieves the various configurations required at boot (e.g. current version, etc.) Dotenv & Environment variables are stored in Puppet and CCM Once all EC2 instances are available, a script runs and applies the configuration on all live instances (in apply mode) Puppet boots, automatically starting the various Node.js processes we need to keep our application alive ( and ) PM2 API Workers For logging metrics, we use a combination of + + . StatsD Graphite Grafana Understanding Electron ⚡ We wanted to experiment with building an app with downloads for every Linux distro, macOS, and Windows, in addition to the web. Fundamentally, this seemed pretty easy: we write code, wrap it in an Electron shell, and release to our desired operating system… It turns out we were wrong. Electron Electron, though powerful, turned out to be a bigger beast than we had anticipated. Building to different distros was especially hard, even with (granted, we had the bad luck of needing to patch electron-builder (and that bug has since been fixed), but that only accounted for some of the pain points we hit). The macOS menu bar had to be just right for the macOS store to accept our app, and performing small tasks with the Electron API, like opening a link in an external browser, turned out to be pretty difficult. Despite the difficulties, our team moved forward with some custom tooling (all visible and open-sourced on ) and we released not only to all of our release targets but to the , too. electron-builder GitHub web Testing in JavaScript 🗳️ JavaScript is still the wild west to a degree. It’s rather un-opinionated, especially if you’re using Express, so we had to roll our own testing framework to get the job done. Our API, which is built with Express, uses a combination of various Node.js modules. Here’s a list of the tools we use for testing: as a testing framework Mocha as an assertion library Chai as our mocking library Sinon as the HTTP mocking library Nock as a module mocking library mock-require as our test coverage tool Istanbul Here’s an actual example of our . Bonus: test runner The combination of test modules we chose our team to move fast with multiple developers working on various feature sets at the same time, without bringing down the API. Frontend 🛠️ React is a phenomenal framework, and in our opinion, has won the battle against other frameworks such as Angular and Ember. Given its updated , it’s perfect for the Winds 2.0 project. MIT license The main stack that we use for Winds 2.0 is pretty straightforward: : Main Stack Create-react-app React (react-redux) Redux React-router Electron Now let’s chat about some of the front-end modules we used to make Winds 2.0 a reality. : Interesting Modules is a nice React interface to the core audio element API React-audio-player for scrolling events to handle automatic pagination React-waypoint for easy OPML file imports React-dropzone for flawless image fallbacks in the event we don’t have an image stored React-image Backend When you’re building a large application, you generally rely on many libraries and tools to increase the code quality, time to market, etc. With that said, Winds too relies on many libraries and tools. Below are several, but not all, that we use: FeedParser is a rather complex Node.js module that in our opinion, is the backbone of the project. It handles most of the inconsistencies found in RSS feeds and spits out a “cleansed” version of the feed. Without this module, we would be writing a lot of if/else statements… and that’s no fun. FeedParser Franc-Min is a language detection module that we utilize for determining the language of a feed. This might sound like a small task; however, it’s in fact, a large part of our personalization engine. For example, we only recommend feeds in English to users who speak English. Same with other languages. Franc-Min Bull helps keep the Winds 2.0 queue structurally sound with the help of . It comes with a super easy API and supports multiple queues, which is a perfect fit for our use-case. Additionally, there are several open-source monitoring tools on their page that provide insight into what is happening behind the scenes. Bull Redis GitHub ES6 enables our team to write minimalist code for a wide range of people. All of the code in Winds 2.0 is 100% JavaScript (with the exception of a handful of bash scripts to help with deploy workflows). The team is currently migrating much of the functionality in the codebase to utilize to reduce the number of lines of code. JavaScript w/ ES6 Async/Await Yarn is absolutely amazing. It’s an incredibly fast package manager built specifically for JavaScript. On top of that, it’s 100% open-source and nearly always available, due to it’s caching mechanisms. We’ve used in the past, and although it works just fine, the team here at Stream prefers Yarn. Yarn npm Axios is a Promise based HTTP client for the browser and Node.js. We actually use it on both the front and back-end for various tasks. For example, all front-end HTTP requests flow through a wrapper to Axios. And for the back-end, we utilize Axios to inspect the size of the file prior to sending them through the parsing process and then off to the database for storage — this ensures large files don’t bring down our worker processes. If you haven’t checked out Axios, you definitely should. Axios Commander is another Node.js module, this time providing full support for building command-line interfaces. Yes, that’s right, Winds has a CLI that we use for various tasks such as testing feeds, dropping RSS feeds, and more! Commander Babel “allows us to use the next generation of JavaScript, today”. Essentially, if a feature, such as imports isn’t available in a particular JavaScript (front-end and/or back-end), we can still use it by leveraging Babel. Babel Express is used to power our . Compared to other frameworks out there, it truly shines when under stress. Honestly speaking, our team has experience with most, if not all, of the JavaScript frameworks and we find Express to be the easiest to work with. It’s regularly maintained, has open-source components, and it’s awesome. 😎 Express API Sentry allows for real-time crash reporting for our back- and front-end. What blows us away is how granular you can get with Sentry. Their features help us identify and debug errors and give us insight on when to fix or rollback. When it comes to firefighting, this tool definitely wins the market. Sentry Algolia provides lightning-fast ( ) search for our application. In 2ms, our users can discover RSS feeds and podcasts to read or listen to. They even have components for frameworks such as React and to make the integration better. This is by far one of our favorite; however, we like them all. 😛 Algolia literally under Angular Stream is a key resource to Winds 2.0, as it provides news feeds and activity streams for our users and even machine learning-based personalization. Without , we would not be able to serve up suggested content to our users as we currently do. Stream Stream MongoDB Atlas is a phenomenal DBaaS, allowing us to worry about acquiring users, while MongoDB worries about uptime. It’s identical to hosting your own cluster, except MongoDB provides you with a dashboard and a URI to connect to. With MongoDB Atlas, there’s no more worrying about cluster health, monitoring, etc. MongoDB Atlas Mongoose is a powerful ODM that allows us to define rich models within our MongoDB environment. Generally speaking, are schemaless (meaning they have or require no form); however, with MongoDB, it’s always a good idea to specify a schema so you can index and organize your data properly. This allows for easy updates and efficient queries. Mongoose NoSQL databases PM2 is a Node.js process manager. It allows us to ensure uptime, and scale processes as we need. The CLI for the project is dead simple, which allowed our team to pick it up on the fly. PM2 Final Thoughts 😁 To get started with Winds 2.0, you can try the web version or download the application . If you’re feeling more adventurous head over to and spin it up locally. here GitHub RSS is in a vicious circle. Winds is a community effort to help turn the tide and . Contributions are always much appreciated. To discuss new features check out the . #ReviveRSS official Slack channel for Winds If you’re curious to know a little more about and how our API works, we have an easy that will walk you through the process of building scalable activity feeds. Stream 5-minute API tour