Hacker Noon Product Update Q1 2020

Welcome to code, where the metrics are made up and the points don’t matter. I’m not #1. I just commit more often. 😛 January and February 2020 were damn fine months for Hacker Noon, and March seems to have become a sh*tshow for most of planet Earth, but we’ve certainly coded a lot. Merging 150+ PRs, modifying thousands of lines of code, we hacked away as fast as our fingers would take us as we gathered in Edwards, CO (before the outbreak) for the Third Semi-Annual Company Advance (we don’t retreat, we advance -- get it?). But what does this translate to? What do those “lines” of code actually mean? Where’s the beef, son? Wanna Be Hacker Noon’s Best and Brightest Writer? Brush Up On Tag Pages The first major feature we deployed in 2020 had to be new tag pages. We (read: mostly Storm Farrell) paginated the tag pages, added all stories to the page (rather than the first 50 stories, as we had before), implemented fast search functionality with Algolia (you can also drill down to search results that include only certain tags), custom built meta description management, and we’ve even been beta testing content relevant sponsorship placements. Storm brought our tag pages up to snuff. Fair warning: this post gets into some gory details on how Hacker Noon’s software platform is designed. You have been warned. A radical shift in Hacker Noon’s CDN architecture Our big release of the quarter was internally dubbed NextGreen: a redesign and re-architecture built with the NextJS framework on the story page, homepage, search page and tagged page. The story page is where the vast majority of our traffic lives, and while we’ve been testing the new design work live for the last three weeks on production, average time on page is up 14% on the story page. Almost as importantly, this new architecture streamlines deployment across our CDN, cloud functions and application, so future iterations can deploy significantly faster. Before the changes, our CDN was based on individual, statically-generated pages for each tag. This meant long regeneration times when a template changed, among other pitfalls. It was fast and cheap though. How did we change things?By using a single template for all tag pages. How? With Google’s CDN, we would normally need a file corresponding to each URL in the proper storage bucket. But in front of the CDN, we have Nginx load balancers, whose original purpose was to simply redirect HTTP to HTTPS (Google’s CDN does not support HTTP forwarding at the time of this writing). We soon expanded these load balancers to handle 301 and 302 redirects, as well as other rewrites. For example, the new tag pages! When you visit a tag page, the Nginx server is configured to rewrite the URL to read from /tagged.html (visiting that page directly may have weird results). This Nginx rewrite preserves the url on the client, so while you see https://hackernoon.com/tagged/blockchain, the server’s still reading from tagged.html, loading a script to read the pathname and fetch the required tag info from Algolia. Why didn’t we go with this approach in the first place? Simply put, the cost of Firebase queries would have been prohibitive. In the near future, though, we’re going to take a new tack, creating a single document for each page that can be read on the cheap and used to render the page in its entirety. So if Storm was working on tag pages, what were Dane and I doing? Hacker Noon’s First Venture Into Storing Content on The Blockchain Dane’s been working on a top-secret super-project for the past several weeks -- more to come on that front, in a separate post. Spoiler alert: it’s a new way to curate and discover. On my end, I went in a different direction: using Mark Nadal’s distributed database, GUN, we made it so you could highlight part of a story and add an inline annotation, a note for the author, a comment, a suggestion, something to provide additional context. This note will be submitted to the author for their approval (or rejection --- the author decides what shows up on their story). An early iteration of Dane’s, showing how annotations can be submitted. The fun thing about annotations, from an implementation perspective, is they used an entirely different database, a distributed database, our first official use of the blockchain in our platform. Unfortunately this experiment is not ready to deploy to all users. GUN will continue to provide load performance insights. Have learned a lot about distributed database. Curious about how text hosting evolves over time. The Next Step in the Evolution of Hacker Noon Once we finished the work (for the most part) on the new tag pages, we started thinking about how to improve our CDN architecture. We initially thought we couldn’t use a traditional framework, or even a JAMstack framework, due to caching and query cost concerns. However, we turned out to be wrong. Storm can explain it best: While the first version of the CDN allowed us to keep things lean by producing plain HTML files for content, it limited us in two ways. Without a framework, maintaining and iterating features is a chore requiring much duplication and limited interactivity, and secondly, any time we deployed a change that affected all instances of a template, we’d have to run scripts to update all the generated content. So after considering a few options, we started experimenting with Next.js which boasts a fair number of benefits for us: Because it’s built on React like our app, the learning curve would not be too steep for our dev teamWe had already built last year’s Noonies on itIt’s flexible Hybrid approach to SSR, static and SSG gives us a lot of control over data fetching and caching, especially as of 9.3.It has an excellent community and documentation with OOTB solutions for almost anythingIt has offline and PWAS support But most importantly, it allows us to deploy changes without having to re-render every post and to publicly cache pages. So between deployments, if one user visits a new story that hasn’t been pre-rendered, it gets rendered, hydrated, and then stored for future visitors. This could have a significant impact on our read usage, which is an important consideration when hosting with Firebase. Hacker Noon 2.1 is now in alpha, but we should be rolling it out sometime over the next few weeks. It comes with a refreshed design, so keep an eye out 😉 This much-improved architecture has been a major boon to frontend iterations. I have a few things in the works around this…but that’s another story. 👀 Among other things, here are some highlights of some of the improvements we deployed in the last quarter: Removed duplicate stories from the homepage and tag pagesRoutable tabs on the author dashboard and the profile page on app.hackernoon.comDark mode fixesEstimated time reading on tag page storiesThe header now slides up when you scroll down and slides down when you scroll upFixed an issue involving duplicate stories being publishedFixed bugs around the headerFixed bugs involving story deletionStory creation fixFixed tagged RSS datesFixed missing tags on the editor dashboardFixed direct routing to tabsDelete button fixLoading bar bugfixTag management UI for adminsAds by tagMeta descriptions and images for social sharing on all pagesInternal Slack notifications when a user updates their profile CTAFix loop in statsFix editor route holeAllow self-publishing for admins and editorsAutosave fixImprove editor routing fixAdd SentryFirestore rules updatesFix admin ad selectionFix ad JSON generationFix brand as author button for editorsFix missing story imagesAlgolia data fixesScheduled story publishingInternal script fixesMoar admin powah [note: literally the PR name]Story stats fixFix stats on referral traffic These, among many other bugfixes whose PRs simply said something along the lines of “fix the thing” 😂Stay tuned as we launch monthly product updates in the coming weeks! 👀Check the #hackernoon tag for updates. Closing Note We have a lot of options to be a better place for technologists to read, write, and publish. I'm looking forward to digging in to how product can improve curation, the writer experience, and building out the reader account experience. Welcome to code, where the metrics are made up and the points don’t matter. I’m not #1. I just commit more often. 😛 Welcome to code, where the metrics are made up and the points don’t matter. I’m not #1. I just commit more often. January and February 2020 were damn fine months for Hacker Noon, and March seems to have become a sh*tshow for most of planet Earth, but we’ve certainly coded a lot. Merging 150+ PRs, modifying thousands of lines of code, we hacked away as fast as our fingers would take us as we gathered in Edwards, CO (before the outbreak) for the Third Semi-Annual Company Advance (we don’t retreat, we advance -- get it?). But what does this translate to? What do those “lines” of code actually mean? Where’s the beef, son? Wanna Be Hacker Noon’s Best and Brightest Writer? Brush Up On Tag Pages The first major feature we deployed in 2020 had to be new tag pages. We (read: mostly Storm Farrell ) paginated the tag pages, added all stories to the page (rather than the first 50 stories, as we had before), implemented fast search functionality with Algolia (you can also drill down to search results that include only certain tags), custom built meta description management, and we’ve even been beta testing content relevant sponsorship placements. Storm brought our tag pages up to snuff. Storm Farrell Fair warning: this post gets into some gory details on how Hacker Noon’s software platform is designed. Fair warning: this post gets into some gory details on how Hacker Noon’s software platform is designed. You have been warned. A radical shift in Hacker Noon’s CDN architecture Our big release of the quarter was internally dubbed NextGreen: a redesign and re-architecture built with the NextJS framework on the story page, homepage, search page and tagged page. The story page is where the vast majority of our traffic lives, and while we’ve been testing the new design work live for the last three weeks on production, average time on page is up 14% on the story page. Almost as importantly, this new architecture streamlines deployment across our CDN, cloud functions and application, so future iterations can deploy significantly faster. Before the changes, our CDN was based on individual, statically-generated pages for each tag. This meant long regeneration times when a template changed, among other pitfalls. It was fast and cheap though. How did we change things? By using a single template for all tag pages. How did we change things? How? How? With Google’s CDN, we would normally need a file corresponding to each URL in the proper storage bucket. But in front of the CDN, we have Nginx load balancers, whose original purpose was to simply redirect HTTP to HTTPS (Google’s CDN does not support HTTP forwarding at the time of this writing). We soon expanded these load balancers to handle 301 and 302 redirects, as well as other rewrites. For example, the new tag pages! When you visit a tag page, the Nginx server is configured to rewrite the URL to read from /tagged.html (visiting that page directly may have weird results). This Nginx rewrite preserves the url on the client, so while you see https://hackernoon.com/tagged/blockchain , the server’s still reading from tagged.html, loading a script to read the pathname and fetch the required tag info from Algolia. https://hackernoon.com/tagged/blockchain Why didn’t we go with this approach in the first place? Why didn’t we go with this approach in the first place? Simply put, the cost of Firebase queries would have been prohibitive. In the near future, though, we’re going to take a new tack, creating a single document for each page that can be read on the cheap and used to render the page in its entirety. So if Storm was working on tag pages, what were Dane and I doing? Hacker Noon’s First Venture Into Storing Content on The Blockchain Dane ’s been working on a top-secret super-project for the past several weeks -- more to come on that front, in a separate post. Spoiler alert: it’s a new way to curate and discover. Dane On my end, I went in a different direction: using Mark Nadal’s distributed database, GUN , we made it so you could highlight part of a story and add an inline annotation, a note for the author, a comment, a suggestion, something to provide additional context. This note will be submitted to the author for their approval (or rejection --- the author decides what shows up on their story). Mark Nadal’s distributed database, GUN An early iteration of Dane’s, showing how annotations can be submitted. An early iteration of Dane’s, showing how annotations can be submitted. The fun thing about annotations, from an implementation perspective, is they used an entirely different database, a distributed database, our first official use of the blockchain in our platform. Unfortunately this experiment is not ready to deploy to all users. GUN will continue to provide load performance insights. Have learned a lot about distributed database. Curious about how text hosting evolves over time. The Next Step in the Evolution of Hacker Noon Once we finished the work (for the most part) on the new tag pages, we started thinking about how to improve our CDN architecture. We initially thought we couldn’t use a traditional framework, or even a JAMstack framework, due to caching and query cost concerns. However, we turned out to be wrong. Storm can explain it best: While the first version of the CDN allowed us to keep things lean by producing plain HTML files for content, it limited us in two ways. Without a framework, maintaining and iterating features is a chore requiring much duplication and limited interactivity, and secondly, any time we deployed a change that affected all instances of a template, we’d have to run scripts to update all the generated content. While the first version of the CDN allowed us to keep things lean by producing plain HTML files for content, it limited us in two ways. Without a framework, maintaining and iterating features is a chore requiring much duplication and limited interactivity, and secondly, any time we deployed a change that affected all instances of a template, we’d have to run scripts to update all the generated content. So after considering a few options, we started experimenting with Next.js which boasts a fair number of benefits for us: So after considering a few options, we started experimenting with Next.js which boasts a fair number of benefits for us: Because it’s built on React like our app, the learning curve would not be too steep for our dev team We had already built last year’s Noonies on it It’s flexible Hybrid approach to SSR, static and SSG gives us a lot of control over data fetching and caching, especially as of 9.3. It has an excellent community and documentation with OOTB solutions for almost anything It has offline and PWAS support Because it’s built on React like our app, the learning curve would not be too steep for our dev team Because it’s built on React like our app, the learning curve would not be too steep for our dev team We had already built last year’s Noonies on it We had already built last year’s Noonies Noonies on it It’s flexible Hybrid approach to SSR, static and SSG gives us a lot of control over data fetching and caching, especially as of 9.3 . It’s flexible Hybrid approach to SSR, static and SSG gives us a lot of control over data fetching and caching, especially as of 9.3 9.3 . It has an excellent community and documentation with OOTB solutions for almost anything It has an excellent community and documentation with OOTB solutions for almost anything It has offline and PWAS support It has offline and PWAS support But most importantly, it allows us to deploy changes without having to re-render every post and to publicly cache pages. So between deployments, if one user visits a new story that hasn’t been pre-rendered, it gets rendered, hydrated, and then stored for future visitors. This could have a significant impact on our read usage, which is an important consideration when hosting with Firebase. Hacker Noon 2.1 is now in alpha, but we should be rolling it out sometime over the next few weeks. It comes with a refreshed design, so keep an eye out 😉 But most importantly, it allows us to deploy changes without having to re-render every post and to publicly cache pages. So between deployments, if one user visits a new story that hasn’t been pre-rendered, it gets rendered, hydrated, and then stored for future visitors. This could have a significant impact on our read usage, which is an important consideration when hosting with Firebase. Hacker Noon 2.1 is now in alpha, but we should be rolling it out sometime over the next few weeks. It comes with a refreshed design, so keep an eye out This much-improved architecture has been a major boon to frontend iterations. I have a few things in the works around this…but that’s another story. 👀 Among other things, here are some highlights of some of the improvements we deployed in the last quarter: Removed duplicate stories from the homepage and tag pages Routable tabs on the author dashboard and the profile page on app.hackernoon.com Dark mode fixes Estimated time reading on tag page stories The header now slides up when you scroll down and slides down when you scroll up Fixed an issue involving duplicate stories being published Fixed bugs around the header Fixed bugs involving story deletion Story creation fix Fixed tagged RSS dates Fixed missing tags on the editor dashboard Fixed direct routing to tabs Delete button fix Loading bar bugfix Tag management UI for admins Ads by tag Meta descriptions and images for social sharing on all pages Internal Slack notifications when a user updates their profile CTA Fix loop in stats Fix editor route hole Allow self-publishing for admins and editors Autosave fix Improve editor routing fix Add Sentry Firestore rules updates Fix admin ad selection Fix ad JSON generation Fix brand as author button for editors Fix missing story images Algolia data fixes Scheduled story publishing Internal script fixes Moar admin powah [note: literally the PR name] Story stats fix Fix stats on referral traffic Removed duplicate stories from the homepage and tag pages Routable tabs on the author dashboard and the profile page on app.hackernoon.com app.hackernoon.com Dark mode fixes Estimated time reading on tag page stories The header now slides up when you scroll down and slides down when you scroll up Fixed an issue involving duplicate stories being published Fixed bugs around the header Fixed bugs involving story deletion Story creation fix Fixed tagged RSS dates Fixed missing tags on the editor dashboard Fixed direct routing to tabs Delete button fix Loading bar bugfix Tag management UI for admins Ads by tag Meta descriptions and images for social sharing on all pages Internal Slack notifications when a user updates their profile CTA profile CTA Fix loop in stats Fix editor route hole Allow self-publishing for admins and editors Autosave fix Improve editor routing fix Add Sentry Firestore rules updates Fix admin ad selection Fix ad JSON generation Fix brand as author button for editors Fix missing story images Algolia data fixes Scheduled story publishing Internal script fixes Moar admin powah [note: literally the PR name] Story stats fix Fix stats on referral traffic These, among many other bugfixes whose PRs simply said something along the lines of “fix the thing” 😂Stay tuned as we launch monthly product updates in the coming weeks! 👀 Check the #hackernoon tag for updates. Check the #hackernoon tag Closing Note We have a lot of options to be a better place for technologists to read, write, and publish. I'm looking forward to digging in to how product can improve curation, the writer experience, and building out the reader account experience.