April Infrastructure Shift Brings May Time On Story Page Lift

Hacker Noon Product Update: Overview of What We Deployed in April 2020

They say "April showers bring May flowers." In our case, April infrastructure changes are going to bring bouquets of benefits for our readers and contributing authors in the months to come.

The Ten Thousand Foot Overview

Let’s look at some data. 

First things first, we have several active repositories right now. First, let’s look at the repository that runs app.hackernoon.com,

hackernoon/green

It looks as though I had an inspired rally centered on the 20th, but in reality, I just deleted

yarn.lock

npm

To answer the last question: Richard Kubina is our newest hire! He’s a part-time developer, a friend of mine, working on a stats overhaul at the moment -- more on that later.

As for whether we’re all just slacking: of course not! Behold, the graph for the hackernoon.com engine,

hackernoon/nextgreen

This graph tells a story. Storm clearly added a lot of code, in this case, a lot of boilerplate, infrastructure, and setup code so that we could code effectively with Next.js. I contributed where I could, but it was mostly bugfixes, corrections for things we had in the system pre-Next. 

Of course, raw numbers don’t tell the full story. The number of commits and lines of code don’t matter. Use cases do. Fixes do. Words do. So, the full story:

When it rains, it pours...

April was largely been a shift for us, from the traditional CDN to using Next.js along with SSG, static site generation. This has provided us with a significant, although not insurmountable, set of challenges. For one, we had a lot of inconsistencies in the data that were exposed by this new system -- we were able to largely stamp those out as pages are generated, making all stories render-able with the same logic.

In other cases, we simply had bugs. Any new system, especially one of this magnitude, is going to involve a good number of bugs to fix. Thankfully, the vast majority of these bugs have been reasonably easy to fix, whether they’re display bugs, unexpected changes in behavior, or simply spots we missed in the transition.

This system has several benefits over the CDN. For example, if we want to display dynamic data, it’s so f*cking easy it’s almost criminal, when you compare it to the old system of static HTML and JS. Not only that, if we want to make changes to the pages, it’s much, much easier to change one component rather than 7 or 8 static templates, and invalidate the cache and regenerate pages on demand, on deploy, instead of manually regenerating all the things. Seriously, we had to run scripts to regenerate the old pages whenever we changed a header or nav or something, it was messy.

There are a couple of less-than-desirable behaviors that we still seek to address with this new architecture. We don’t always have the control over the cache that we need, so invalidation on-demand is a major desire for the dev team. Thankfully, a feature known as "incremental builds" will provide a solution to at least one issue, where story updates don’t display immediately (don’t fret -- we have your changes and they’ll show up in ~15 minutes for now).

...and sometimes, it hails...

More importantly, we have a different, slightly concerning caching problem.

We started out with Firebase hosting. It auto-scales, it handled Next.js well enough, but it wasn’t built for Next.js. Moreover, we started to have deploy problems as our bundle hit the 100MB limit (pre-generating pages, mostly). It was like trying to fit a square peg into a round hole.

So we moved a sizeable portion of our reader experience to Zeit, or as it’s now known, Vercel. In short, the bee’s motherf*cking knees. You connect it to Github (or Gitlab or Bitbucket) and bingo bango, deploys are automatic. Just merge your branch into master. Oh, and when you create a PR, it creates a preview branch with your code deployed for everyone to try.

There’s still a significant issue or two, that we’re still working out with NextJS - it's an issue I’d rather not get into the details on out of respect for Vercel and appreciation for their prompt responses. Just a kink to work out, that’s all. Overall, 10/10 would recommend.

Ah, but I ramble.

...but eventually, the flowers bloom.

Overall, the move to Vercel has been relatively smooth -- very few issues, as opposed to the overall CDN-to-Next.js move (even that wasn't too bad). Other than that, a few things we’ve been working on:

• A complete overhaul of stats infrastructure, using a new caching layer to drastically reduce load.
• A more solid backup system than what Firestore and Google Cloud Platform offer by default (yes, we still host our data, storage, and just about everything else on GCP).
• Improving authentication on the internal API.
• Making it possible to test the API locally (we went a surprisingly long time without this).
• ...I’d put a description of our other PRs here, but let’s just say, they all start with the word "fix"

tl;dr

So what's in it for you?

• Your reading experience is going to be improved more often, and faster, than in the past, thanks to the new hackernoon.com infrastructure. Time on page on the story page is up 14% in the preliminary data.
• We have fixed a metric sh*t ton of bugs. The site is becoming more stable by the day -- we're getting to a point where we can spend less time stabilizing and more time adding essential functionality.
• Your stats will be returning soon, I promise! Richard's hard at work, making them way faster and more reliable than they've ever been. Stay tuned!
• Oh, did I mention you can change your own handle now? 😁
• In general, development is going to be happening faster, and new features and fixes will be deployed more often.

In the next month or three, keep your eyes peeled for several incoming features, especially features serving our dear readers. 👀 ‘Til then, hack on!