Our mission at StackShare is to make developers more productive by helping them learn about technology through the people and companies they trust

StackShare

stackshare

Our favorite thing to do around here is find the hottest and most exciting new developer tools and share them with you. If you , you’re getting the inside scoop every week! But let’s take a moment at the end of another month to celebrate the most popular new tools added in all of February. subscribe to our weekly newsletter These are the top 10 new tools added to the StackShare database in February 2018. #1: Haiku Build and design cross-platform UIs and Animations “Haiku lets you design & build interactive, visually stunning UIs,” says creator Zack Brown. “Haiku has a lot in common with After Effects and Flash: it’s a timeline-powered animation tool that supports interactions through code — but unlike After Effects it’s intended for designing UIs, and unlike Flash, it’s built on open standards and an open source, hackable renderer & file format. Haiku is all about bringing design and code closer together.” | View Website View Tool Profile #2: Rekit Toolkit for building scalable web applications with React, Redux and React-router Nate Wang created Rekit because he felt creating a React Redux app required too much verbose boilerplate code. “Rekit helps to generate them automatically and refactor them easily,” he says. “It’s very important to keep a web app project readable, understandable and maintainable when it scales. Rekit suggests an opinionated pattern to organize files and folders in a scalable way and provides IDE/tools to help you always follow the approach. ” How does Rekit compare to similar tools, and how is it better? Wang says “Rekit studio is an IDE that understands your web projects; it knows features, components, actions and routing rules so it can show and manage them in a meaningful way, while other IDEs are more like just code editors which only know folders, files.” | View Website View Tool Profile #3: Docusaurus Easy to maintain open source documentation websites Facebook engineer Joel Marcey his team created Docusaurus for a number of reasons: says To put the focus on writing good documentation instead of worrying about the infrastructure of a website. To provide features that many of our open source websites need like blog support, search and versioning. To make it easy to push updates, new features, and bug fixes to everyone all at once. And, finally, to provide a consistent look and feel across our all our open source projects. “At Facebook,” says Marcey, “Docusaurus allows us to quickly get different projects up and running with documentation websites, especially for teams who don’t have much experience with web development or primarily want a basic site to showcase their project.” | View Website View Tool Profile #4: Proton Native A React environment for cross platform native desktop app Gustav Hansen initially created Proton Native “because I had to create a complex GUI for another project, and I didn’t understand why something so simple became so hard. I had close to a 1K lines of code for something that was repetitive and not too complex. I tried creating it in PyQt, but it became a mess.” Hansen had already used React a lot and loved the workflow, so “I decided to see if anyone had done something like that for desktop. I couldn’t find anything, so I made my own. It quickly morphed into a response to the bloat of Electron and the split in developing for desktop.” | View Website View Tool Profile #5: Apache OpenWhisk A serverless, open-source cloud platform Apache’s OpenWhisk homepage describes the tool this way: “As a developer there’s no need to manage the servers that run your code. Apache OpenWhisk operates and scales your application for you. Spend your time innovating, not configuring infrastructure. Plus, you only pay for what your app uses and needs at that moment.” | View Website View Tool Profile #6: Maze Create missions that testers will perform on your InVision prototype, and learn how your product’s design can be improved with zero lines of code. “Too many businesses are wasting insane amounts of time and money building features just to realize that the design simply doesn’t work,” says Maze creator Jonathan Widawkski. “This leads to frustration as teams end up rebuilding features instead of implementing new ones. “We realised that there was a way to collect data much earlier in the process: performing quantitative user testing at the prototype phase to iterate quickly and effectively until your design is proven. This is why we created Maze: an affordable analytics and testing solution built on top of your InVision’s prototype.” Further, he says, “Maze is based on the tools designers already use (InVision) and doesn’t require testers to install anything.” | Visit Website Visit Tool Profile #7: Blazor An experimental web UI framework using C#/Razor and HTML, running in the browser via WebAssembly Blazor creator Steve Sanderson of Microsoft built Blazor as a web framework that utilizes Web Assembly to run .NET in any browser. “Blazor is currently an experimental project and in the early stages of development…there isn’t a download available at this time.” But “if you’re keen, you can clone the repo, build it, and run the tests.” | View Website View Tool Profile #8: dotmesh Capture, organize, and share application states using a git-like CLI Creator Luke Marsden says “We created dotmesh because data is missing from the circle of control: while managing code changes with version control, automated tests, and CI is obvious to modern developers, and managing cloud resources with declarative tools like Terraform is obvious to modern DevOps, the data in your app is left in the cold. With dotmesh, you can capture the state of multiple microservices’ data stores in a single commit and branch and push/pull them between stages of the software lifecycle such as dev, CI, staging and (soon) production.” While intentionally similar to Git in terms of interaction patterns and command (to ensure it’s familiar to developers), Marsden says dotmesh is different in three important ways. “Firstly it supports efficiently snapshotting large databases (e.g. a Postgres database), which Git just isn’t set up to support; secondly it supports a feature called subdots, which enable more than one data store to be captured in a single commit so that you can capture the state of an entire app in one go, even if it’s a microservices app with polyglot persistence; and thirdly it supports clustering out of the box, meaning that it can operate either on a single docker host (e.g. Docker for Mac on your laptop) all the way up to your production Kubernetes cluster (e.g. on GKE).” Find out more in dotmesh’s and “Please give us feedback on Slack; as a new startup we’re very keen to hear feedback from users!” docs | View Website View Tool Profile #9: PopcornNotify Send simple emails and text messages from one API “It’s common for developers to want to send notifications from their code,” says PopcornNotify creator Jason Strauss. “There are tools for sending emails and SMS, but they take a bit of configuration: buying phone numbers, verifying domains, etc. “Those configurations are really important when communicating with your users, but unimportant for internal messages, like ‘The server is down’ or ‘Your long script has finished running.’ I built PopcornNotify because I have several small projects and internal tools, and I wanted a simple, automated way to send myself messages from them.” | View Website View Tool Profile #10: Airtap Test your JavaScript in 800+ browsers “Airtap is different than other cross browser test runners in its simplicity and ability to easily run your test suite in many browsers without having them installed locally. It lets you iterate quickly during development and provide good browser coverage before release without worrying about missing a supported browser.” | View Website View Tool Profile If you use and love any of these new tools, be sure to add them to your Stack and to your feed! Now let’s gear up for another month of new tools! News Originally published at stackshare.io .

Top 10 New Developer Tools of February

React was named our in the Application & Data Category, and the buzz just keeps growing in 2018. At StackShare we keep our finger on of the hottest tools in the industry, and it seems like something new and exciting is happening in the React world every day! Top Developer Tool of 2017 the pulse Here are the top 10 February React articles from across the web, as hand-curated by the StackShare team using data from . StackShare News Want to keep up? Just add React to and you’ll be as on top of it as we are. your customized newsfeed #1: What’s new in React 16.3(.0-alpha) Bartosz Szczeciński’s Medium post outlining all the “biggest, most interesting changes” in the newest alpha release of React has nearly 8k claps. | View Reddit Discussion View More React News #2: Some great React components, libraries, and tools for 2018 “A curated collection of useful and popular React.js UI components, tools and libraries for building your next amazing app in 2018.” | View Reddit Discussion View More React News #3: Top 10 ReactJS Open Source Projects of February (by Mybridge) This new monthly series by Mybridge AI ranks the top open source React projects on GitHub based on a variety of factors. Some of these projects were so cool we had to add them to the StackShare database (check out and ). Docusaurus React Easy State | View Reddit Discussion View More React News #4: React’s new Context API Ken C. Dodds wrote this post about the new Context API in React, and why you shouldn’t (necessarily) be afraid to use it. | View Reddit Discussion View More React News #5: Build a Progressive Web App using React Aman Mittal’s guide walks you step by step through building a simple PWA with React. | View Reddit Discussion View More React News #6: Introducing Rekit Studio: a real IDE for React and Redux development Nate Wang announced the stable release of Rekit Studio, “a complete IDE for React, Redux, and React Router development.” And yes, of course it’s ! If you’ve used it, add it to your stack now! on StackShare | View Reddit Discussion View More React News #7: Free Course — Learn React by building an app to help you find the best movie to watch If you’re not already on the React train, this is a great program for those of us who learn by doing. Build a simple app with CodeCamp’s free tutorial. | View Reddit Discussion View React News #8: How I Shipped a Neural Network on iOS with CoreML, PyTorch, and React Native From Stefano J. Attardi: “This is the story of how I trained a simple neural network to solve a well-defined yet novel challenge in a real iOS app.” | View Hacker News Discussion View React News #9: React patterns you should know João Miguel Cunha reminds us: “React is just JavaScript!” This post goes through some common problems and gives examples of how you can use patterns to solve them. | View Reddit Discussion View React News #10: Developing Games with React, Redux, and SVG — Part 2 This blog post by Auth0 is the second in a series that walks you through developing a React game called Aliens, Go Home! | View Reddit Discussion View React News React Tweet of the Month More! Get all the React news, every day, at . StackShare News StackShare News is the best place to stay up to date every single day on what’s happening with your favorite developer tools. It’s the newsfeed customize, so you can keep up with all the tools you use and love (and none of the ones you don’t). Check it out and build your own ! you here Originally published at stackshare.io .

React Roundup: Top 10 Articles in February

Top 50 Developer Tools of 2017 🏆

Originally posted on StackShare About Opendoor Unless you’re in San Francisco or New York, selling your home is a giant headache that typically lasts three months. removes the headache — go online, ask for an offer, answer a few questions and we’ll buy your home directly from you. We’ll take it from there and deal with selling the home to another buyer while you can go on with your life. Opendoor Right now we operate in Phoenix, Dallas, and Las Vegas. We’ve completed over 4,800 real estate transactions — over $1B in homes. For a company about to turn 3 years old, it’s pretty crazy how far we’ve come. There’s a lot that goes into one real estate transaction. First, there’s what you might consider our core engineering challenge: making an accurate offer on each home. If we offer too much, we’ll lose money and go out of business; if we offer too little, we’ll seem like scammers and offend our customers. After we buy the home, we’ll work with contractors to do any necessary repairs and touch-ups, then put the home on the market and find a buyer. Since we own every home, we can do clever things like putting smart locks on all the doors and offering all-day open houses. I’m a frontend engineer, and mainly like to work on the consumer-facing website. I’m currently working on improving the experience for first-time home buyers. The process can be really scary for people who don’t know anything about real estate. Engineering Organization Our team is split between product engineering and data science: the tools used by each team are different enough that the teams work in separate code bases. Of course, the resulting product has to be well-integrated, and the product team pulls a lot of data from data science APIs. This coordination is tricky to get right; Kevin Teh from the data science team wrote about it in some detail in . a recent post At first, we split the product team into “customer-facing” and “internal tools” groups. It was nice to have all the frontend engineers on the same team, but we noticed that some projects didn’t have clear owners. For example, our buyer support team uses some internal tools we’ve built. Should those tools be developed by the “internal tools” team, or is support part of the customer experience? Now the team is split into cross-functional teams based around parts of the business. The Seller team handles people selling to us; the Homes team handles renovations and inventory; and the Buyer team puts our homes on the market and finds buyers. As we grow, the lines between teams often get blurry, so we expect that the structure will always be evolving. It’s common for engineers to move between teams, including between the product and data science teams. Product Architecture We started in 2014 with a monolith and frontend, both of which were good ways to move fast while we were very small. Ruby on Rails Angular The MVP of our customer-facing product was a multi-page form where you could enter information about your home to get an offer, but that was just the tip of the iceberg. We had to build internal tools to help our team correctly price homes and manage the transaction process. We used and to build out those tools; the main goal was to add features quickly, without fiddling around with CSS — in fact, without requiring any frontend experience at all. Angular Bootstrap We use as our webserver, and for our database — one big benefit is the extension for location data. runs our asynchronous jobs with support from . shows up everywhere in our internal tools. We use to build our frontend apps, and serve them using the Rails Asset Pipeline. Puma Postgres PostGIS Sidekiq Redis Elasticsearch Webpack We use to store photos of our homes, as well as most of the icons and illustrations around our site. We mainly use Imgix’s auto-resizing feature, so we never lose track of our original images, but can later load images of appropriate size for each context on the frontend. Imgix Monolith to Microservices Where appropriate, we try to break isolated logic out into microservices. For example, we’re working on a service which calculates our projected costs and fees. Our cost structure changes frequently, and we want to estimate how policy changes might affect our fees. This code wasn’t a great fit for the Rails app because we wanted it to be accessible to our analysts and data scientists as well. We’ve now split this logic out into its own service. It uses a version-history-aware computation graph to calculate and back-test our internal costs, and (soon!) will come with its own frontend to visualize those calculations. React Our data science stack is also a fully separate set of services, so there’s a lot of inter-app communication going on. To let these services authenticate to one another, we use an app called . Opendoor engineer Dan Neighman wrote and open-sourced Paladin, and explains why it’s helpful in . Authentication is based on JWTs provided by and . Elixir Paladin this blog post Warden Guardian Data Science Architecture I’ve always found data science at Opendoor interesting because it’s not the “grab as much data as you possibly can, then process it at huge scale” problem I’m used to hearing about. To find the price of a house, you look at nearby homes that sold recently, then squeeze as much information out of that data as you possibly can by comparing it to what you know about the market as a whole. . Our co-founder Ian Wong has a more in-depth talk here We can group most of the data science work into several core areas: Ingesting and organizing data from a variety of sources Training machine learning models to predict home value and market risk Quantifying and mitigating various forms of risk, including macroeconomic and individual house-liquidity Collecting information in a data warehouse to empower the analytics team For data ingestion, we pull from a variety of sources (like tax record and assessor data). We dump most of this data into an database. We also transform and normalize everything at this phase — we’re importing dirty data from sources that often conflict. goes into more detail on how we merge data for a given address. RDS Postgres This blog post For our machine learning model, we use with building blocks from , , and . We use for routing/handling requests. We use to build images and for deployment and scaling. Our system lets us describe a model as a JSON configuration, and once deployed, the system automatically grabs the required features, trains the model, and evaluates how well the model did against performance metrics. . Python SqlAlchemy scikit-learn Pandas Flask Docker Kubernetes This automation lets us iterate really fast We’re starting to use for feature fetching and processing. Other companies often use Spark and Hadoop for this, but we need support for more complex parallel algorithms. Dask’s describes this perfectly: Dask comparison to PySpark post Dask is lighter weight and is easier to integrate into existing code and hardware. If your problems vary beyond typical ETL + SQL and you want to add flexible parallelism to existing solutions then dask may be a good fit, especially if you are already using Python and associated libraries like NumPy and Pandas. The final piece of our data science architecture is the Data Warehouse, which we use to collect analytics data from everywhere we can. For a long time we used a nightly to move data from each service’s database directly into a home-built Data Warehouse. We recently migrated to Google’s instead. BigQuery is faster, and lets us fit more data into each query, but the killer feature is that it’s serverless. We have many people running queries at “peak hours”, and don’t want things to slow down just because we have a preallocated number of servers available. pg_dump Postgres BigQuery High-Tech Open Houses Since Opendoor actually owns all the houses we sell, we can be creative about how we show them to potential buyers. Typically, if you want to see a house for sale, you have to call the listing agent and schedule a time. We realized early on that we could make open houses way more convenient by installing automatic locks on our doors so the homes could be accessed at any time. For version 0 of the project, we literally posted our VP of Product’s phone number on the doors of all our houses — buyers would call in, and he’d tell them the unlock code. For version 1, we added so we could automatically send unlock codes over SMS. For version 2, we built a mobile app. Twilio Customers expect a good mobile experience these days, but our all-day open house feature made it twice as important. You can use the app to find nearby homes as you’re driving around, and explore them on a whim — a huge improvement from the traditional process! We built our app in . A major part of that choice was pragmatic — our team had a lot of experience with web technologies, and almost no experience with native technologies. We also wanted to support both iPhone and Android from early on, and React Native let us do that (we released our app for iPhone first, and adding Android only took an extra couple weeks). React Native Not everyone wants to install an app, so it’s still possible to access our homes via SMS. We’ve added a few security mechanisms — one worth mentioning is , which lets us quickly run identity verification using phone numbers. For riskier numbers, we disable the automatic entry system and have our support team call the customer to collect their information. Blockscore Tools and Workflows We manage our repositories and do code reviews on . All code is reviewed by at least one other engineer, but once it’s in the branch, it’s assumed to be ready to deploy. If you want to deploy your code, you can do it in three steps: GitHub master ./bin/deploy staging Check your work on staging ./bin/deploy production This takes 10–15 minutes in total. We’ve worked hard to automate the process so we can move fast as a team. We use for hosting, and run automated tests on . bots report what’s being deployed. Heroku CircleCI Slack There are a lot of external services we rely on heavily. To run through them briefly: and for emails; and for calls and customer service; for online contract signing; and for system monitoring; for error reporting. Help Scout Dyn Talkdesk Twilio HelloSign New Relic Papertrail Sentry For analytics, we’ve used a lot of tools: for the web, for mobile, for retroactive event tracking. We mainly use for digging into that data and making dashboards. Mixpanel Amplitude Heap Looker Joining Opendoor Engineering Opendoor has a very entrepreneurial, pragmatic culture: Engineers here typically talk with customers, understand their needs, and take the initiative on projects. We’re big on ownership and empowering others, and are aggressively anti-snark. We’re looking for engineers of all backgrounds: it doesn’t matter what languages you work with now, we’re sure you’ll ramp up fast. Find out more about Opendoor jobs on or on our . StackShare careers site Huge thanks to Kevin Teh, Mike Chen, Nelson Ray, Ian Wong, and Alexey Komissarouk for their help putting together this post.

The Stack That Helped Opendoor Buy and Sell Over $1B in Homes

In this blog post, I will guide you through deploying an Amazon Clone application on Aptible, showcasing how this platform can simplify and secure the process. I'll cover essential steps, from creating an Aptible account to provisioning a PostgreSQL database, setting up the backend and frontend applications, configuring environment variables, and deploying your applications. Following this comprehensive guide, developers can gain valuable experience deploying web applications on Aptible, a secure and scalable cloud infrastructure platform. The provided instructions and code snippets will simplify the deployment process and help you understand how to harness cloud infrastructure effectively. Features of the Amazon Clone Built using Next.js Styled with Tailwind CSS Backend generated using Amplication State Management with Zustand Typescript for type safety Login Signup using JWT Tokens API calls using Axios Recharts for charts Next UI for exceptional prebuilt components React Pro Sidebar for a prebuilt excellent sidebar. Admin Panel Table Pagination Functionalities of the Amazon Clone Categories CRUD (Create/Read/Update/Delete) Products CRUD (Create/Read/Update/Delete) Order Management Dashboard for Admin Admin Login User Login & Signup User view products User search products The user adds products to the cart User place order And many more features that you can watch from the video below This blog post is a part of my YouTube video. Check out the fantastic video below: https://www.youtube.com/watch?v=JVVJcFGV5p4&embedable=true Introducing Aptible: Simplify and Secure Your Application Deployment Before diving into the details of deploying your Amazon Clone application on Aptible, let's take a moment to introduce Aptible and understand why it's an excellent choice for hosting your web applications. Aptible is a cloud platform designed to simplify and streamline the Deployment and management of web applications, particularly for developers and teams prioritising security and compliance. With Aptible, you can focus on building and scaling your applications while leaving the complexities of infrastructure, security, and compliance to the platform. One of Aptible's standout features is its commitment to security. It provides robust security controls and compliance certifications, making it an ideal choice for applications that handle sensitive data or need to meet regulatory requirements. Aptible takes care of security updates, data encryption, and access controls, allowing you to focus on your application code. Now, let's get started. Step 1: Create a new Aptible Account. Head to Aptible and click the login button to create a new account and sign up. After signup, you will be redirected to the Aptible Environments page. Step 2: Choose Environment Now, you will see your created environments on this page. What are Environments? Environments live within Stacks and provide logical isolation of resources. Environments on the same Stack share networks and underlying hosts. Now, choose your environment and click on it. This is your environment screen. From here, you can manage your apps and databases, see your activity, backups, and many more settings. Step 3: Provision a new PostgreSQL Database We have used the PostgreSQL Database for our Amazon clone, so let's provision a new Database on Aptible. Click on the Database Tab. Now, you will be able to see the Database screen. You can manage and provision new databases from here in your Aptible Environment. Now click on the New Database button to start creating a new database. You can see the UI change to Choose a database selection as soon as you click it. Now, we need to select a database. Click on the database selection dropdown. It will show you all the supported databases by Aptible. Currently, Aptible supports the following Database types: CouchDB PostgreSQL MongoDB Redis Elasticsearch InfluxDB MySQL SFTP RabbitMQ You can check their supported database page to see if they have added more databases. Now, from the list of databases, select PostgreSQL 14. You can see a field as a database handle when you select it. You can name the database of your choice using this field. We are going to name it. . amazon-db Now click on the button. Save changes Now, Aptible will start provisioning your database. Wait for it to complete. While the database is being provisioned, let's move on to the next step, as this will run in the background and be completed in a few minutes. Step 4: Setup Amazon Clone backend on Aptible Now, let's deploy our backend app on Aptible. Head over to the tab and click on . Apps New App Now, you will see the new app creation window. In this window, you need to provide the name of your app. We are going to name it . amazon-backend Now click on the . Create App button Now, Aptible will start creating your app inside the selected environment. Now, you will need to paste your SSH key in this step. If you want to generate a new SSH key or want to learn how Aptible uses your key, visit . Aptible SSH Docs After pasting your SSH key, click on . This will store your SSH key in Aptible so that you don't have to repeat the process the next time you create an app in Aptible. Save key Now we need to push our code to Aptible's git repo. For this step, we would be required to create a inside the root of our server code. Dockerfile You need to copy the following snippet inside the Dockerfile # Development Dockerfile
# This Dockerfile is intended for development use and includes development dependencies and tools.

# Use a Node.js base image with development tools
FROM node:16.0.0 AS development

# Create a directory where the application will be built
WORKDIR /app

# Copy over the dependency manifests, both the package.json
# and the package-lock.json are copied over
COPY package*.json ./

# Install packages and their dependencies
RUN npm install


# Copy over the Prisma schema
COPY prisma/schema.prisma ./prisma/

# Generate the Prisma client based on the schema
RUN npm run prisma:generate

# Copy over the code base
COPY . .

RUN npx prisma db push
# Create the bundle of the application
RUN npm run build

# Expose a specific port on the Docker container for development purposes
ENV PORT=3000
EXPOSE ${PORT}

# Start the development server using the previously built application
CMD ["npm", "start"] Now open up your terminal and write the following steps: Step 1: Create a git repo git init Step 2: Stage all files git add . Step 3: Commit the changes git commit -m "Initial Commit for backend code. After you have committed your changes, you need to copy the Aptible remote url and add it to your repo. git remote add aptible git@beta.aptible.com:youtube-a32/amazon-backend.git After this step, we must push the code to the Aptible's scan branch. Copy the scan command from Aptible and paste it into your terminal. git push aptible main:aptible-scan Now, this will push our code to Aptible's git repository. Now, in Aptible, it will show you this screen. We can configure our environment variables and DB for the app from here. Let's select a DB. Click on the button. Connect Existing Database Now select the that we created in the previous step and name the environment variable as . amazon-db db_url Now click on the button to start the deployment process. Save and Deploy This will start the deployment process. This process may take a certain time. Wait for it to complete. Now, our app has been deployed successfully. Now, we need to run the docker command. Select the cmd and click on the button. Create Endpoint Now, Aptible will start provisioning your endpoint. Our endpoint was successfully provisioned. Now, let's head over to the endpoint. And you can see that we have successfully deployed our Node server to Aptible. Now, let's do the same for our Next app. Step 5: Deploy Amazon Clone Frontend to Aptible Deploying the front end is quite simple. We will follow all the previous steps and deploy our app to Aptible. You need to replace the file in your next app with the code below. next.config.js /** @type {import('next').NextConfig} */
const nextConfig = {
  reactStrictMode: false,
  env: {
    NEXT_PUBLIC_CLOUDINARY_CLOUD_NAME: "dctahvizk",
    NEXT_PUBLIC_API:"app-61711.on-aptible.com"
  },
  images: {
    domains: ["res.cloudinary.com"],
  },
  // typescript: {
  //   ignoreBuildErrors: true,
  // },
};

module.exports = nextConfig; Here, we have added the environment variable. NEXT_PUBLIC_API Make sure that you replace it with your endpoint. Now, create a new Aptible app and name it . amazon-next-app Before moving forward, let's add the for your next app. Dockerfile # Use an official Node.js runtime as the base image
FROM node:18

# Set the working directory inside the container
WORKDIR /app

# Copy package.json and package-lock.json (if available)
COPY package*.json ./

# Install dependencies
RUN npm install

# Copy the rest of your app's source code
COPY . .

# Build the production version of the app
RUN npm run build

# Expose the port your Next.js app will run on
EXPOSE 3000

# Start your Next.js app
CMD ["npm", "start"] Create the at the root level of your app and paste this code. Dockerfile Now open up your terminal and write the following steps: Step 1: Create a git repo git init Step 2: Stage all files git add . Step 3: Commit the changes git commit -m "Initial Commit for frontend code. After you have committed your changes, you need to copy the Aptible remote url and add it to your repo. git remote add aptible git@beta.aptible.com:youtube-a32/amazon-next-app.git After this step, we must push the code to the Aptible's scan branch. Copy the scan command from Aptible and paste it into your terminal. git push aptible main:aptible-scan Our code was pushed to Aptible. Now, let's deploy it. Click on the button. Save & Deploy Our Deployment was successful. Now, let's run the docker cmd and provision an endpoint. Now, our endpoint was successfully provisioned. Open the endpoint in your browser. And you can see that our Next app was successfully deployed on Aptible. Conclusion In conclusion, this blog post has provided a comprehensive guide on deploying an Amazon Clone application on the Aptible platform. We covered essential steps such as creating an Aptible account, provisioning a PostgreSQL database, setting up the backend and frontend applications, configuring environment variables, and deploying the applications. By following this guide, developers can gain valuable experience in deploying web applications on Aptible, a secure and scalable cloud infrastructure platform. The provided instructions and code snippets simplify the deployment process and help developers understand how to leverage cloud infrastructure effectively. Aptible offers a robust environment for hosting web applications, making it an excellent choice for developers looking to deploy and manage their projects securely. This tutorial equips developers with the knowledge and practical skills needed to take advantage of Aptible's capabilities for deploying web applications.

Deploying a Full Stack Amazon Clone with Next.js, Tailwind CSS, Zustand, and Nest.js on Aptible

148 Stories To Learn About Php

The craze for learning programming languages is very high. These days people have one question in their mind and that is which programming language should one prefer in 2022 and why? So The Themeselection is sharing here the Top 10 Demanding programming languages 2021 . In this detailed article, you will get to know what a specific language offers you and how it is getting into the trends. All these mentioned striking programming languages hold high scopes for the upcoming era. We have made this detailed list of the top 10 striking programming languages 2022 as per the recent survey. For programmers, coders, and other developers seeking development work, it’s never been more important to stay updated on the programming languages being used throughout the world. Although some of these languages share key similarities others require an entirely different level of understanding. Besides, having a deep knowledge of the right programming language can help land the right job or find work in a particular field. If we look at pay-scales then the following were some highest paid languages last year. Credit: Stackoverflow Here we are going to discuss the top 10 striking programming languages for 2022. The list contains the following languages: Rust TypeScript Python Kotlin Go Solidity Dart C/C++/C# Swift Javascript Sources Used: Stackoverflow Stackshare Hired Github Google trends Note: We have prepared this list after researching and observing the performance, demand, and usability of each in 2021. Although, this data may change as data can’t be static. Some languages in the list may seem new to you but we have included them because of the scope they possess. In the end, it depends on users as everyone has different interests and tastes. Also, we did not single out the first and last place. Today, the choice of language depends on the scope. Therefore, the main indicator is the frequency of use in different areas. Now, let’s head to the list…!! Rust Rust is a multi-paradigm programming language designed for performance and safety, especially safe concurrency. While Rust is syntactically similar to C++, one thing that it can guarantee is memory safety by using a borrow checker to validate references. As per the survey, Rust has been for consecutive four years. Impressive, isn’t it? This huge popularity shows that those who have had the opportunity to use Rust have fallen in love with it. Stack Overflow’s most loved language Credit: Stackoverflow Besides, Rust allows you to store data on the stack or on the heap and determines at compile time when memory is no longer needed and can be cleaned up. This allows efficient usage of memory as well as more performant memory access. Companies using Rust: Dropbox Coursera Figma Discord Advantages: Better memory safety due to the compiler Easier concurrency due to the data ownership model that prevents data races Zero-cost abstractions So, both beginners and experienced programmers can kickstart with Rust. In the way it is used, the language is close to established alternatives. TypeScript Typescript is a well-known name. Since its release in 2014, it is growing rapidly and maintaining its place in the list of top 10 striking programming languages 2021. It is developed by Microsoft. Well, it has emerged in response to growing dissatisfaction with JavaScript. Soon after its arrival, It impressed the Google team so much that they began working on developing TypeScript, instead of developing a new language. Credit: Stackoverflow Basically, TypeScript was used as the main programming language for the Angular2 + SPA structure. Even, Vue.js team is also using typescript as they announced in 2018 to develop new versions of its framework. The reason behind its popularity is, it is easy for developers to write and maintain codes. Furthermore, it offers a complete description of each component of the code and can be used for developing large applications with a strict syntax and fewer errors. Companies are diverting towards typescript. Following are some companies that are using typescript. Slack Vox Media medium Accenture ViaVarejo Advantages: Class and Module Support. Static Type-checking. ES6 Feature Support. Clear Library API Definition. Build-in Support for JavaScript Packaging. Syntax Similarity to Backend Languages (Java, Scala, .Net) In addition, it is well-structured and easy to learn. Its extended toolbox makes application development quick. With such benefits it offers, TypeScript is expected to supersede JS in 2020–2021, making it one of the most sought-after programming languages in the future. Nowadays many of the JS frameworks provide support for typescript. You can check and which are made using typescript . Modern Admin - Angular 10+ Bootstrap 4 Admin Dashboard Template Apex Angular Admi Template Python Image credit: Teahub Python is trending nowadays like never before..!! It is the biggest competitor to Javascript. Especially because of the rising trend of AI..!! Most often, Python is used in Data Science, Machine Learning, Web Development, Computer Science Education, Computer Vision, and Image Processing, Game Development. Besides, it is considered one of the easiest software languages to learn and master. Also, Python is undoubtedly the best starting language for beginners. Often named one of the world’s most popular programming languages, Python powers the backend stacks of many startups and is essential in the growing fields of data science and AI Credit: Octoverse.github According to many reports and ratings, is rapidly gaining popularity and has every chance of moving JavaScript from the first place. Many biggies are using python for example: Python Instagram Spotify Amazon Facebook Advantages: Very easy to use and learn Emphases on code readability Wide-ranging library support Open-source programming language Have a good community support Can scale complex applications Build a prototype Credit: Google trends As you can see how python has surpassed javascript in the past 5 years. Besides, it has a very wide range of applications like Web & Desktop Development, Network Servers, Machine Learning, GUI Based Desktop applications, In Enterprise and Business and Data Science, etc. Kotlin was developed by the Jet Brains team in an attempt to find “something better than Java.” Many experts claim that the goal has been achieved. Like Go, Kotlin cannot be called the trendiest programming language. But statistics show that demand is growing. Kotlin The effortless interoperation between Java and Kotlin makes Android development faster and more enjoyable. Since Kotlin addresses the major issues that surfaced in Java, developers have rewritten several Java apps in Kotlin. According to the , is one of the five most popular programming languages in the world. It beat competitors such as Java and Python, giving in only to Go and Scala in its field. Hired rating Kotlin Credit: Hired Companies using KOTLIN Coursera Uber Pinterest Post mates Advantage: A good compiler An Efficient programming language Provides an improved run-time performance Absence of raw types. In addition, the number of Kotlin users in the Github community is growing incredibly fast. According to a report from Octoverse, Kotlin ranks 4th among fast-growing programming languages. Go Well, is the least known of all the programming languages, but that doesn’t mean it is something you neglect..!! It was developed by Google in 2007 for APIs and web applications. Go In the past few years, Go has become one of the fastest-growing programming languages. According to Some programmers, it is a great alternative to Python. Credit: Google Trends As per the latest Report, Go has been going down. Although, it is necessary to draw your attention that Go has been maintaining its position consistently on the list. There may be some break downs but it has grown up to a level where it can’t be ignored…!! Credit: Hired Go is Ideal for minimal web applications, APIs, and web servers. Besides, it includes some features like C-like semantics, memory management, high performance, and modern data structure support. You can use it for system and network programming, big data, machine learning, audio, and video editing, and more. Here are some notable firms that have done so: Google Apple Facebook Docker The New York Times BBC Advantages: It Is Fast Easy To Learn Well-Scaled Comprehensive Programming Tools So, according to the trends and reports, despite its fall in the current time it is wise to keep it in check as it has maintained its popularity over the years. It is possible that Go can rise again. Solidity Now, we would like to pay attention to . Well, this programming language is not in the list of the best version of Stackoverflow or Octoverse. Moreover, it is not into the top 10 Pypl and TIOBE. Similarly, even in hired Solidity did not become one of the most sought. Solidity So, “Why Solidity is on the list?”. As we said at the beginning, one of the major factors is the scope. When interviewing Stackoverflow regarding Blockchain development, most often respondents mentioned this one. It’s a well know fact that demand for over the past year has grown rapidly. By almost by 517%. This technology is completely new because it is not yet very popular. But the benefits of the blockchain and statistics indicate that in the coming years the situation will change radically. Blockchain specialists Blockchain technology has emerged as one of the most promising services in recent years. It holds great potential to transform the workings of the financial sector. Living in a highly digitalized world, a range of large and medium-sized enterprises are exploring blockchain technology services in order to make a place in this competitive market. Thus, the demand for qualified solidity developers will increase. As it is the best for Blockchain today. Therefore, in 2021 the demand for specialists will be very large. This programming language is supported by the fast-growing EOS platform. It is Ethereum’s main competitor. So, If you plan to connect your life with the blockchain, then you must learn exactly Solidity and C++. Dart: Dart is a client-optimized programming language for apps on multiple platforms. It is developed by Google and is used to build mobile, desktop, server, and web applications. Dart is an object-oriented, class-based, garbage-collected language with C-style syntax. Dart can compile to either native code or JavaScript. Before Google announced Flutter, Dart was nowhere to be found. However, since the announcement of Flutter in Google I/O, Dart has got drastic attention among mobile developers as an alternative to React Native. The developers who didn’t like JavaScript as the programming language found Dart as an alternative option. As big companies like Google, , etc adopted Flutter, the popularity of Dart raised considerably. Alibaba Credit: Stackoverflow Advantages: It is open-source. Dart is really flexible Easy to learn It has great tooling support It ensures productivity Dart can be compiled both and . AOT JIT Companies using Dart: Google Useinsider Pickio Agora Besides, Almost every major Text Editor and IDE has great support for the Dart language. You can use heavy IDEs like Webstorm, IntelliJ IDEA, and Android Studio, or use simple editors like VS Code, Sublime Text, Vim, Emacs, Atom, etc. So you are free to choose whatever editor you are comfortable with. C/C++/C Yes, these are three But, as we said, the main focus is the scope. C/C++/C# are equally often used for game development. If you are confused about where to start refer to different programming languages. As we know, the gaming industry is very huge and is gaining more and more momentum every year. Besides, Demand for the quality of games is growing, therefore, qualified developers are always needed. As per the report, about 50% of the world’s population spends time to video games every day. Not only in Gamin Industry but Embedded applications such as robotics, 3d printing, Operating systems, drivers, Desktop applications such as graphics suites, desktop word processors, etc... uses this language. Also, all three of these programming languages occupy fairly high positions in all ratings. So, in the Hired list, they were in 13–15 places Credit: Hired Companies Using C# Delivery Hero Microsoft Accenture ViaVarejo Stack Overflow Companies using C++: Evernote LinkedIn Microsoft Opera NASA Facebook Companies using C: Twitch GitHub Telegram Messenger MasterCard Lime Swift Swift is a general-purpose, multi-paradigm, compiled programming language developed by Apple Inc. and the open-source community, first released in 2014 As name sands for, It is a powerful and intuitive programming language. It is useful for macOS, iOS, watchOS, tvOS, and beyond. Besides, writing Swift code is interactive & fun, the syntax is concise yet expressive. In addition, Swift includes modern features developers love. Swift code is safe by design, yet also produces software that runs lightning-fast. Credit: Stackoverflow As you can see in the report above, it is in the list of top 10 most loved languages 2020..!! And why not? It offers some amazing advantages as follows: Rapid Development Process Easy to Read and Write Requires a lot less code Improved Performance and Safety Easy to Integrate with Objective-C Companies using Swift: Lyft Uber Slack Artsy Credit: Armadsen Furthermore, since Swift is promoted by Apple, its popularity and community support are increasing. In fact, a study of the top 110 apps on the app store showed that 42% of apps are already using Swift. If only the 79 non-game apps are considered, the results are that 57% are using Swift, while 43% are not. Although the Swift community is growing fast, still it is significantly smaller as compared to any other open-source language. According to the latest StackOverflow Developer Survey, only 6.6 percent of the 87.354 respondents use Swift. JavaScript JavaScript is the most famous programming language today. Despite the flaws that triggered the creation of add-ons such as CoffeeScript, Flow, and TypeScript, JS is still the most sought after in many areas. It is widely known for adding interactive elements to web applications and browsers. Most front-end developers prefer JS. However, the Node.js runtime provides significantly more features than just front-end development Credit: octoverse is the most popular language, according to a Stackoverflow survey. The same survey showed that 66% of respondents like to work with JS. 17.8% of respondents want to master JS. JavaScript Credit: Hired As you can see, Javascript is still ruling the software development world..!! Many big companies are using Javascript, for example: Microsoft PayPal Groupon Google And why not..!!? It offers some amazing advantages as follows. Advantage: Runs instantly inside the web browser No compilation requirement Offers a very responsive interface to a website Highly multipurpose languages Simplistic implementation You can check Sneat Bootstrap 5 HTML Admin Template, and both are made using Javascript. Materio Vuetify VueJS Admin Template It can also be used to create animations and control the web interface. Another benefit is its support for all modern browsers. JavaScript can also be used for backend development with the help of Node.js. Conclusion: So, this was the list of the Top 10 Striking Programming languages 2021. As we have covered the 10 significant languages here, there are several that we have not mentioned. But the demand for relevant specialists will be very high. Among them are: Scala Ruby Java Objective-C PHP Well, despite there being some on the list and some are not, it is not necessary that you just go with the flow..!! You can choose the language you feel is more relevant and find it easy to work with. It will be beneficial to learn any of these languages. So, pick your favorite and kickstart your journey…!! About Themeselection: ThemeSelection, provide selected high quality, modern design, professional and easy-to-use premium and free , and UI Kits. bootstrap admin templates You can check which is one of the best free bootstrap admin templates. Chameleon Free Bootstrap Admin Template Also published at https://dev.to/theme_selection/top-10-striking-programming-languages-2021-4d2h

Top 10 Programming Languages To Learn in 2022

New tech terms and lingo are constantly popping up in the development world. It’s hard to keep up! People also love to share and debate about their favorite tech stacks and tools / frameworks. That said, there is no ultimate right or wrong answer when it comes to choosing a tech stack because it really depends on the specific use case it’s meant to solve for. Today, let’s take a look at the HERN stack. Ever heard of it? It’s a new “buzz word” as they say, and I thought it might be helpful to provide some clarity. The HERN stack stands for HarperDB, Express, React, and Node, the four technologies that make up this particular stack, and they’re all written using JavaScript. This is a full-stack solution that follows a common 3-tier architecture (front end, application, and database). As mentioned, there are numerous stacks out there and different factors play a role in the decision around which is right for your specific end goal. Remember that you have the freedom to choose and try different stacks! To make sure we’re all on the same page, Stackshare defines a tech stack as “the set of technologies an organization uses to build a web or mobile application. It is a combination of programming languages, frameworks, libraries, patterns, servers, UI/UX solutions, software, and tools used by its developers.” So as you can see, the combinations and possibilities are endless! What Makes up the HERN Stack? Distributed SQL / NoSQL hybrid database accessed via a REST API. HarperDB : Open source, back end web application framework for Node.js, designed for building web applications and APIs. Express.js : Open source, front end, JavaScript library for building user interfaces or UI components. React.js : Open source, cross-platform, back-end JavaScript runtime environment that executes JavaScript code outside a web browser. Node.js : There are several reasons that these technologies work well together and serve as a great full-stack solution for projects like app development and web development. There are also variants of this stack that would work just as well, such as swapping out and incorporating frameworks like Vue, Angular, or . HEAN, HEVN, HFRN… don’t be afraid to get creative and choose what technologies make the most sense for you! Fastify How Does the HERN Stack Work? Like most tech stacks, this one has a frontend, backend, and database tier. The HERN stack allows you to write everything in Javascript, ultimately simplifying and streamlining the development process. Each component of the HERN stack provides a unique benefit, and data flows naturally through each of the three tiers, enabling rapid development and simple upkeep. ( !) Fun fact - HarperDB is actually the only database that was built from the ground up in Node.js, which you can read more about here Frontend React serves as the frontend or web tier of the HERN stack. Among the many reasons React is , it enables faster development, simplifies coding and integration, and comes with a strong community. The HarperDB was written in React. React has improved the quality of front end development and allowed us to quickly make changes and improvements to our application. taking over frontend development Management Studio HarperDB also provides a great , if that’s your style, or you could make calls directly to HarperDB via our built-in, easy to use API. Either way, you can query data directly from within your client-side application if you choose. are a great place to manage your API calls and how responses are handled within your app. React SDK React Hooks Backend Node.js and Express.js serve as the backend, server, or application tier of the HERN stack. Express essentially operates inside a Node server, as the server-side framework. is a minimal and flexible framework, with numerous HTTP utility methods and middleware at your disposal. Express The idea here is that you build your custom API functions (GET, POST, etc.) with Express to be called by the React front end. These functions then make calls from the server to HarperDB to access and modify the database. You can utilize the HarperDB , which natively supports promises and callbacks. The library makes HarperDB interactions feel more native to JavaScript by exposing each HarperDB operation as a function on the client object. Node.js client Database If you need to store or transport any type of data, which you probably do, you will also need a database. Why not choose the quickest, easiest, and most intuitive database out there? HarperDB provides full document store capability plus enterprise grade ACID-compliant SQL, effectively allowing you to run SQL on JSON. The database is accessed via a single endpoint REST API using SQL and NoSQL operations. HarperDB has simplified much of the work when it comes to database installation, configuration, and administration. It’s really a great option for developers of any skill level. Check out if you’re building in the cloud, or you can also do a local . HarperDB Cloud install on your server Is HERN Your Next Full-Stack Solution? You can essentially build anything with the HERN stack, and it’s especially great for JavaScript developers and projects with lots of JSON data. With a dynamic schema, HarperDB was designed to ingest and handle any type of data at scale. You no longer need to pick a database based on the structure of your data. HarperDB provides the flexibility to work with the frameworks of your choice, and ability to connect to your favorite reporting or analysis tools. Scaling is easy with HarperDB’s clustering and replication features, and our data model, built on top of LMDB, is enabling us to break records on both read and write speeds. HarperDB works extremely well with Express, React, and Node, and with this tech stack you can maximize your time writing code, and work on your project instead of debugging and other administration tasks. Don’t get stuck in your old ways! Try the new HERN, HEAN, or HEVN stack and let us know what you think. We would love to hear what you’re working on! Also published at https://dev.to/harperdb/hern-stack-comin-in-hot-pa6

Along Came HERN Stack

Writing good documentation is crucial. While working on Appwrite, documentation is one of the essential parts of our development life-cycle. When we are building tools designed for developers, our docs are where developers first meet our product. Complex, unclear, or unorganized documentation site can drive developers away even if your product is great. It doesn’t worth a lot if your code is awesome or neat if no-one can use it. With these thoughts on our heads, we set a list of go-to rules regarding the development of our docs site. It was extremely important for us to build docs that developers will love and find easy to both use and understand. In this post I will do my best to share some of the principles we came up with while trying to treat our docs with the same patience and dedication as we would normally treat our source-code. 1. Design & Aesthetics Your design is the first thing developers will see when they try to deep dive into your docs. Actually, this is the first thing they see even before they read a single line of text you wrote. You don’t have to be a design guru, but you can make sure your text formatting is clear and clean. Use colors with good contrast, have enough spacing and make sure your font is readable. It’s always nice to validate that your design complies with W3C accessibility guidelines. Dark mode is also a feature developers highly appreciate, especially if they like to wear a dark sweatshirts and seat in the dark :). 2. Start at the Beginning Think what are the most required, and un-skippable steps that a user must make to start getting value from your project. I like to count the number of steps necessary to accomplish each action and to try and optimize that number. Remember, your docs should fulfill the need of developers checking your project for the first time and the ones that are working with it daily. Finding the right balance is crucial. 3. Understand your Target Audience Some dev projects might target different types of developers. Have a good separation of content for different target audiences. For example, we have different integrations for different development platforms. We have integration for web, mobile, native, and backend developers. Each audience has its own getting started guide and path. We try to require minimum preliminary knowledge to align our learning curve as much as possible. 4. Headlines & Anchors Headlines and link anchors are a great way to structure your data. Anchors will help developers to share your content easily, and headlines will allow developers that are already working with your project to jump exactly to the point where they left your docs in their last visit. 5. Think What’s Next Don’t expect anyone to understand your product as well as you do. Developers’ time is precious. At the end of each phase or tutorial, think about what will be the best next step for your users to take. Always think about how you can help guide them through the perfect flow in order to master your tool. This can be another doc, section, or even some external resources that might be relevant. As with every other product, try to drive action and help your users to complete multiple steps. 6. Examples, Examples, and Examples Again, the developer time is precious. Don’t waste it. Many developers will often try and skip your long text and jump right into your code examples to save time and find out whether your project is relevant for them. Make sure to have plenty of examples for every step in your flow. Make sure your examples are simple and short as possible. No-one wants to start debugging your 300 line code or waste 15 minutes to find one single function. You developed a tool for developers to use, and its goal is probably to save them some time. Writing complex code examples will just do the opposite. Write cohesive code examples and let each example accomplish a goal. It’s also nice to have a link to a complete working script in case your code examples are highly dependent on each other. You can host the complete script on a 3rd party site like or to make your docs lighter. GitHub Codepen 7. Be Transparent Developers are smart, treat them like that. I am writing code for over 20 years, and I don’t like magic happening around me. I do want to understand what is happening in the tools I use. I do not necessarily want to know every bit of detail, but I would love to have a basic understating of the concepts under the hood. Having your code open-source is one great way to gain transparency and trust. The other is to share information about your architecture, tool stack, and design tradeoffs. At Appwrite, we have our available on GitHub, our explained in our repo, and our tool stack shared on . source-code design spec stackshare.io 8. Don’t push Advanced Topics at the Start Simplicity is king. This phrase is right for both software and content. Make sure your docs are simple and don’t require a huge learning curve or preliminary knowledge that is avoidable. Have a section for advanced topics, or at least make sure not to mix content for beginner and ninja-level developers together. Avoid throwing the heavy guns right at the start. 9. Always Iterate Your docs will never be perfect. It might be due to lack of time, lack of initial understanding of who your audience is, or because your product shifted direction multiple times. No matter the reason, it’s ok as long as you understand it, and you are constantly working to improve them. Talk with your community, ask and answer questions, always doubt, and ask yourself what could have you done better to avoid a user issue. You can’t be cohesive and yet answer every question at the same time, but you can always strive to improve and look for a better balance. Conclusion Writing docs is not as fun as writing code, but it’s sure as important. Your docs are where the developer will meet your project and decide if it fits their needs. Good docs can help write software faster just as much as good source-code or a great API will. You should definitely dedicate the time to structure your docs to (almost) perfection. Learn More Appwrite is an that is driven by its community. You can learn more about Appwrite features, API, and integration by visiting Appwrite or . You can also follow Appwrite news and announcements on our or accounts. You can also chat with us on our . open-source backend server official website Github repository Twitter Facebook Discord server About Me My name is Eldad Fux. I am an Entrepreneur, Software Architect, open-source enthusiastic, and the creator of . You can follow me on my Twitter account: or connect with me via Linkedin: . appwrite.io https://twitter.com/eldadfux https://www.linkedin.com/in/eldadfux/

Your Docs Are the Gateway to Your Devtool

If Kubernetes is the pilot that steers the ship, then Helm Charts are the navigational maps that guide the journey correctly. We have seen this happening over the years in the world of software development, one tool comes with an idea, serves the purpose, and becomes obsolete in some years, and then one more tool gets introduced that replaces the old approach, and this journey continues. Similarly, today is the era where Kubernetes is looked upon as the king of container orchestration. Backed by Google, Kubernetes has grown leaps and bounds by increasing its reach and community to every part of the world. But, as we all know, with high power and popularity, Kubernetes comes with some complexity. To mitigate this complexity that Kubernetes poses, we have Helm. What is Helm? How does it work with Kubernetes? We will discuss all this further in the article. What is Kubernetes & how it works? is an open-source container orchestration tool that is backed by Google and is becoming too popular in the software industry because of the benefits that it provides. Kubernetes Think, for example, you are building an application using containers, you go ahead and deploy the application, the application becomes too popular and to keep up with the increased popularity, you need to scale the resources and now, instead of a few containers, you have to deal with hundreds of containers. That becomes a mess, you need a simple way to automate the process, and this is where Kubernetes comes to the rescue. Image credits: Knoldus .Inc In Kubernetes, there is a master node and multiple worker nodes, each worker node can handle multiple pods. Pods are just a bunch of containers clustered together as a working unit. You can start designing your applications using pods. Once your pods are ready, you can specify pod definitions to the master node, and how many you want to deploy. From this point, Kubernetes is in control. It takes the pods and deploys them to the worker nods. If a worker node goes down, Kubernetes starts new pods on a functioning worker node. This makes the process of managing the containers easy and simple. It makes it easy to build and add more features and improving the application to attain greater customer satisfaction. Know more in this video in detail: What is Kubernetes Kubernetes was originally designed by Google and then it was later donated to the (CNCF). Cloud Native Computing Foundation Why is Kubernetes so powerful? Horizontal scaling Self-healing Load balancing and service discovery Automated rollouts and rollbacks Storage orchestration Batch execution Configuration management What is Helm & how it works? is a tool for Kubernetes that helps you install and manage applications. In terms of architecture, Helm has two sides, there is a client-side, which is the Helm command-line tool (helm CLI), and there is also a service side, which is called Tiller. Helm Image credits: On Slideshare by Codefresh The helm CLI is what you execute and run in your local command-line environment. It uses a templating engine to generate Kubernetes YAML from some source templates that you set up in Helm. Once the YAML has been generated, it then sends those requests to the Tiller that’s running on your Kubernetes cluster. Tiller then executes updates inside your Kubernetes cluster to make sure it is up to date with what you needed based on the chart, and the tiller will make sure that gets released and will be added to the helm history so that you can rollback to it in the future. Helm terminologies: Image credits: DigitalOcean Packaged Kubernetes (k8s) resources (metadata) Chart: Parametrize and support multiple environments Values: Registry of consumable charts, enables to share and reuse configurations Chart Repository: A deployed instance of a chart Release: Templates are Kubernetes manifest files that describe the resources you want to have on the cluster. They help us control operations during deployment. To deploy your applications using helm, you need to package your applications into a chart (directory with some files in a specific structure) Templates: The top-level directory is the name of your chart and, below that, you have chart.yaml, which stores metadata and version information for your chart. There is values.yaml to store default configuration values. requirements.yaml lets you specify dependencies like MongoDB or Postgres, and then there is Charts subdirectory where dependencies and packages are stored. Finally, there is templates directory, this is where we store source templates that are fed into the helm templating engine. Let’s not get into the technical details so we can stick to our topic. Helm and Kubernetes to help you deploy with ease The software firms are moving away from a monolithic architectural pattern to the microservices pattern, where simple software units work individually to perform a specific function. Different pieces are loosely copulated and are made to communicate with each other without breaking anything. Adopting to microservices boosts knowledge across different teams to work on a single piece of application with full responsibility. The microservices are essential; they create ease to manage, update, and scale-up applications individually, unlike monolithic applications. Helm is a game-changer here, it has altered the way developers define, store, and manage the server-side applications. Helm, the package manager, simplifies the management of applications, and the implementation of microservices. Helm wraps the microservices and all the dependencies together. Helm with Kubernetes is the most popular choice for managing containers on the cloud. The automatic deployment, ease of use, stability, and portability are Kubernetes key features. It includes multiple storage APIs, health check of container, systematic upgrades, and manual or automatic scaling. Kubernetes solves a lot of problems, Makes sure all your containers are up and running Helps in service discovery Resource isolation and utilization efficiencies Prevent’s vendor lock-in problems Since everything is declarative in YALM, Devs can do Ops as well Kubernetes has big support from the community to help you solve your tech problems …..it comes with its own complex problems. Kubernetes is powerful but Kubernetes can be highly overwhelming and might become complex with all the objects that you need to handle - objects, config maps, services, pods, persistent volumes, and the list goes on. All this becomes too complex when you multiply all these objects with the number of releases that you need to manage. How do you manage all these complex activities? There comes Helm to the rescue. Helm helps you package all of that complexity into one simple application and what you can configure. Functions of Helm: Install software Automatically install software dependencies Upgrade software Configure software deployment Fetch packages from repositories Helm makes it easy to start using Kubernetes with real applications. has become the basic toolset for any DevOps specialist. This guide about offers some best practices to structure and specify your Helm charts that will help K8s deliver your container applications smoothly. Helm provides several CI/CD pipeline integration hooks, we can configure actions to occur; for example, before installation begins, or after an upgrade has finished. You can even set health checks for Helm to run and verify a deployment has completed successfully. Kubernetes + Helm duo Helm charts best practices Let’s look this example, Buffer chose Helm for its deployments For more than six years, Buffer, a social media management platform, had a monolithic architecture. In 2016, they decided to move to service-oriented architecture. They started using Docker for their development environments, and they needed a solution that can work well with containers. After assessing several tools in the market like Mesosphere and Amazon ECS, the Buffer crew settled on Kubernetes. They liked Kubernetes because of the problems that it solves with containers, by its relative maturity, large community, and cloud-provider-agnosticism. The Buffer engineering team began the migration process by rebuilding a piece of the monolith's functionality into its own, self-contained service. The project intended to set a remarkable example for other microservices at Buffer moving forward. After the success of this project, they looked towards breaking out different pieces of the monolith & making them separate microservices units. Two years later, Buffer is running three k8s clusters with over 140 services. They faced several along the way, but then they're solving by them moving onto Helm package manager that allows developers to define Charts to describe k8s configuration making deployments easy. challenges with Kubernetes deployments [Source credits for above Buffer's story: ] Stackshare , a cloud-native data science lab, had a challenge handling infrastructure as most of the folks were data scientists or mathematicians, they wanted to make these folks work painlessly. One more good example is Neptune After going through several alternatives, the team decided to use Kubernetes as a layer of abstraction that separates data scientists from the low-level infrastructure problems. Kubernetes started smoothly handling the fluctuating amount of resources. Along with Kubernetes, the team started embracing Helm templates to reduce the time needed to run new machines and start an experiment. [Source credits for above Neptune's story: ] CodiLime No doubt, Kubernetes has become a go-to solution for managing and running your application in Docker containers. Like we discussed in the article, with high power & popularity comes greater complexity. The Kubernetes YAML files & working with Kubernetes as a whole can be a bit hard. Helm is a tool that manages these deployments for you. It makes it easy to do versioning and templating of Kubernetes file. It allows you to define dependencies between deployable components to make the deployments and releases smooth as silk. Rimas Mocevicius, one of the co-founders of Helm recently wrote an article in which he talks about the and some major changes introduced in Helm v3. Helm v3 Beta Features Rimas, in his own words, these are the changes and enhancements in the Helm v3 beta features, Tiller has been removed. Release information is now stored in the Namespace. helm init command has been removed. The Helm home directory has been located off user's home directory. The stable repository is no longer added by default. The helm search command has been refactored to have sub-commands that can search the local repositories and the . Source credits: (I have taken Rimas Mocevicius’s permission to talk about it here) Helm Hub Rimusz Blog Removal of Tiller has been the most significant part in the Helm v3 and for the people who hated it and it has become a part of the history. Migration from v2 to v3 website should help you see how to migrate Helm v2 to v3 with use cases - when you have Helm v2 and v3 managing the same cluster, this case is when you want to move out of Helm v2 gradually and do not require v3 to manage any releases deployed by v2. In other case, migrating Helm v2 to Helm v3 where you want Helm v3 to manage existing Helm v2 releases. If you have any questions related to migration from v2 to v3, take a look at the official . This official guide from Helm Helm FAQ page Helm helps to optimize Kubernetes deployments with ease. Helm has become a popular tool in the Kubernetes ecosystem, gives all developers a way of building packages (known as charts) of related Kubernetes objects that can be deployed in a cohesive way to a cluster. It also enables parameterizing these packages, so they can be reused in different contexts and deployed to the different environments that the services they provide might be needed in.

Kubernetes and Helm: A Deadly Combo to Help You Deploy with Ease

, short for Remote Dictionary Server, is a BSD-licensed, open-source in-memory key-value data structure store written in C language by Salvatore Sanfillipo and was first released on May 10, 2009. Depending on how it is configured, Redis can act like a database, a cache or a . It’s important to note that Redis is a NoSQL database system. This implies that unlike SQL (Structured Query Language) driven database systems like MySQL, PostgreSQL, and Oracle, Redis does not store data in well-defined database schemas which constitute tables, rows, and columns. Instead, Redis stores data in data structures which makes it very flexible to use. In this blog, we outline the by the different core data structure types. Redis message broker top Redis use cases Data Structures in Redis Let’s have  a look at some of the data types that Redis supports. In Redis, we have strings, lists, sets, sorted sets, and hashes, which we are going to cover in this article. Additionally, we have other data types such as , hyperloglogs and geospatial indexes with radius queries and streams. While there are some Redis GUI tools written by the Redis community, the command line is by far the most important client, unlike popular SQL databases users which often , for instance, phpMyAdmin for MySQL and PgAdmin for PostgreSQL. bitmaps prefer GUI management systems Let us take a closer look at the data types that exist in Redis. Redis Strings Redis Strings are the most basic type of Redis value leveraged by all other data structure types, and are quite similar to strings in other programming languages such as Java or Python. Strings, which can contain any data type, are considered binary safe and have a maximum length of 512MB. Here are a couple useful commands for Redis strings: To store a string ‘john’ under a key such as ‘student’ in Redis, run the command: SET “student” “john” To retrieve the string, use the GET command as shown: GET “student” To delete the string contained in the key use the DEL command: DEL “student” Redis Strings Use Cases Many websites leverage Redis Strings to create a to speed up their website experience by caching HTML fragments or pages. Since data is stored temporarily in the RAM, this attribute makes Redis a perfect choice as a session cache. It is able to temporarily store user-specific data, for instance, items stored in a shopping cart in an online store, which is crucial in that your users do not lose their data in the event they log out or lose connection. Session Cache: session cache Any application that deals with traffic congestion, messaging, data gathering, job management, or packer routing should consider a , as this can help you manage your queue size by rate of arrival and departure for resource distribution. Queues: Redis Queue A lesser known use case for Redis Strings is the real-time metering for . This allows SaaS platforms that bill based on actual usage to meter their customers activity, such as in the telecommunications industry where they may charge for text messages or minutes. Usage & Metered Billing: consumption-based pricing models Redis Lists Lists contain strings that are sorted by their insertion order. With Redis Lists, you can add items to the head or tail of the lists, which is very useful for queueing jobs. If there are more urgent jobs you require to be executed, these can be pushed in front of other lower priority jobs in the queue. We would use the command to insert an element at the head, or left of the string, and the command to insert at the tail, or right of our string. Let’s look at an example: LPUSH RPUSH LPUSH list x   # now the list is "x" LPUSH list y   # now the list is "y","x" RPUSH list z   # now the list is "y","x","z" (notice how the ‘z’ element was added to the end of the list by RPUSH command) Redis List Use Cases Social platforms like Twitter use Redis Lists to populate their timelines or homepage feeds, and can customize the top of their feeds with trending tweets or stories. Social Networking Sites: Create news feeds from custom sources where you can pull the latest updates and allow interested followers to subscribe to your RSS feed. RSS Feeds: Forums like Reddit and other voting platforms leverage Redis Lists to add articles to the leaderboard and sort by most voted entries. Leaderboards: Learn how to build your own Twitter feed in our blog post. Caching tweets using Node.js, Redis and Socket.io Redis Sets are powerful data types that support powerful operations like intersections and unions. They are not in any order and are usually used when you want to perform an audit and see relationships between various variables. Sets are reasonably fast, and regardless of the number of elements you have stored, it will take the same time to add or remove items in a set. Furthermore, sets do not allow duplicate keys or duplicate members, so a key added multiple times in a set will simply be ignored. This is driven by a function called which avoids duplication of multiple similar entries. The can be employed when checking unique values, and can also for scheduling jobs running in the background, including cron jobs which are automated scripts. Redis Sets SADD SADD attribute These are particularly helpful for analyzing real-time customer behavior for your online shopping site. For instance, if you’re running an online clothing store, Redis Sorted Sets employ relationship matching technique such as unions, intersections, and subtractions (commonly applied in Venn diagrams) to give an accurate picture of customer behavior. You can retrieve data on shopping patterns between genders, which clothes products sell more the most, and which hours record the highest sales. Redis Sets Use Cases Many use Redis Sets to analyze customer behavior, such as searches or purchases for a specific product category or subcategory. For example, an online bookstore owner can find out how many customers purchased medical books in Psychology. Analyzing Ecommerce Sales: online stores Redis Sets are a great tool for developers who want to analyze all of the IP addresses that visited a specific website page or blog post, and to be able to ignore all of the duplicates for unique visitors with their SADD function. IP Address Tracking: For any app that collects user input, it’s a good idea to implement for inappropriate words, and you can do this with Redis Sets by adding words you’d like to filter to a SET key and the SADD command. Inappropriate Content Filtering: content filtering Sorted Sets As the name suggests, are a collection of strings that assign an order to your elements, and are one of the most advanced data structures in Redis. These are similar to Redis Sets, only that Sets have no order while Sorted Sets associate every member with a . Sorted Sets are known for being very fast, as you can return ordered lists and access elements in the shortest time possible. Redis Sorted Sets score Redis Sorted Sets Use Cases Many Q&A platforms like and Quora use Redis Sorted Sets to rank the highest voted answers for each proposed question to ensure the best quality content is listed at the top of the page. Q&A Platforms: Stack Overflow leverage Redis Sorted Sets to maintain their high score lists, as scores can be repeated, but the strings which contain the unique user details cannot. Gaming App Scoreboards: Online gaming apps Redis Sorted Sets are a great tool for a , as you can associate a score to rank the priority of a task in your queue. For any task that does not have a score noted, you can use the WEIGHTS option to a default of 1. Task Scheduling Service: task scheduling service The Redis geo indexing API uses a Sorted Set for the technique which allows you to index locations based on latitude and longitude, turning multi dimensional data into linear data. Geo Hashing: Geo Hash Redis Hashes are maps between string fields and string values. This is the go-to data type if you need to essentially create a container of unique fields and their values to represent objects. Hashes allow you to store a decent amount of fields, up to (more than 4 billion), while taking up very little space. You should use Redis Hashes whenever possible, as you can use a small Redis instance to store millions of objects. You can use basic operations, such as get, set, exists, in addition to many advanced operations. Redis Hashes 232 – 1 field-value pairs hash command Redis Hashes Use Cases Many web applications use Redis Hashes for their user profiles, as they can use a single hash for all the user fields, such as name, surname, email, password, etc. User Profiles: Social platforms like leverage Redis Hashes to map all the archived user photos or posts back to a single user. The hashing mechanism allows them to look up and return values very quickly, fit the data in memory, and leverage data persistence in the event one of their servers dies. User Posts: Instagram can leverage Redis hashes to record and store their product and sales metrics in a way that guarantees solid separation between each tenant, as hashes can be encoded efficiently in a very small memory space. Storing Multi-Tenant Metrics: Multi-tenant applications Who uses Redis? Redis has found a huge market share across the travel and hospitality, community forums, social media, SaaS, and ecommerce industries to name just a few. Some of the leading companies include Pinterest, Uber, Slack, Airbnb, Twitter, and Stack Overflow. Here are some stats on Redis popularity today: who use Redis 4,107 companies reported using Redis on StackShare 8,759 developers stated using Redis on StackShare 38,094 users have starred Redis GitHub #8 ranked database on with a score of 144.08 DB-Engines

Most Common Redis Use Cases by Core Data Structures

A detailed guide to becoming a technical product manager with links to relevant resources and courses. Good PMs add to engineering teams. You don’t have to know how to code well in order to understand the fundamentals of what happens under the hood of a web browser or how to query an endpoint or even design a database data structure. Understanding the fundamentals of , at the very least, helps you understand what engineers on your team are saying when you are having a conversation with them. And understanding the technical feasibility of a feature or product is an advantage. API programming In fact, it helps you to do your work better. Take for example you are working on an API driven product that you need to write functional specifications for. You need to understand the logic for fetching data from a hosted API server without the frontend experience coupled with it and other concepts like protocols, data formats, authentication, API design, and real-time communication. It’s okay to ask for help from your engineers when working in such an environment but then again it helps if you can anticipate what will be happening in the actual implementation. I find that being able to understand the basic principles of programming really helps inform product thinking and working with engineers more straightforward. If you can clearly communicate the requirements of a feature, you will end up much closer to being on the same page with your team from day one. Remember, it’s not important to be good at coding. You are in no way required to become a developer yourself. You are merely learning fundamentals and concepts so you can more successfully communicate with your team and also be aware of and assess organizational bias in technical decisions. From my experience with interfacing with engineering teams, I will try and take you through the most basic concepts and tools that will help you improve technically and the resources to guide you. 1. Software architecture for product managers. I will be sure to remind you again that this is the role of your engineering team, so there is no need to be thorough with your understanding. However, during product ideation, you should be able to translate business requirements to technical specifications for your team and scoping software architecture is a discussion product managers should be able to contribute to. You should be able to identify the stakeholders required on the project. It is important to identify domain architects with the right expertise that affect the product. For example, if you have already identified a microservice architecture with components like data pipelines and messaging. You would typically require an engineer that has experience with Kafka technology or other publish-subscribe systems. Another reason is choosing the technologies for the implementation of each component and the connections between the components. Now that you have a team to work on the project, you will need to plan on implementation and the resources required for the project. And discussions such as when to use open source or to go proprietary will come up, you will need to navigate a sea of issues like the cost of the technology, what do you need the software to do? what attributes should the technology have? where does it fit into the rest of your infrastructure, and business objectives? These questions guide the rest of the discovery and recommendation process. You can start by reading books that introduces the concepts and best practices of software architecture — how a software system is structured and how that system’s elements are meant to interact. is a good book with practical case studies that can help you get started. Another great site with a detailed resource is by Chris Richardson. Software Architecture in Practice (3rd Edition) by Len Bass, Paul Clements, and Rick Kazman Microservices.io Tweetzine from explaining browser/server comms (Cont….) Mariko Kosaka 2. Learn more about your company’s stack. You should be able to identify what programming stack your company uses or what stack is suitable for a product and even contribute to such discussions. Again it is the responsibility of your engineering team to select a suitable stack for projects but it helps that you are able to know the stack preference of your company and the reason behind the chosen stack and how well it works for your particular use case. For example, if you are working on a data visualization and processing product, you should be able to identify stack (Elastic search, Log-stash, and Kibana) or any other stack as the preferable technology and identify engineers with experience using such technologies. ELK Understand the basic concept of the language your company uses and their limitations and why it is the preferred language. For example, if your company’s frontend is React, it is important that you understand what components, element, and properties are in React. All programming languages have a to help you get started and also a resource to learn concepts of the language step by step. practical tutorial This from CodeMentor for beginners explains in detail how Javascript properties work along with functions, loops, variables, threads, and more. You can also get started with this introductory and friendly to Javascript functions from Mozilla. Javascript tutorial article There are several engineering blogs from teams like , , , and that have cultivated a culture of sharing best practices and their experience with a particular language that you can follow. Be curious in your search for knowledge. Uber Airbnb Netflix, Facebook Google 3. Follow up on new trends in engineering and best practices. Since software involves assigning names to abstract concepts, there is an enormous amount of jargon in tech and it can be hard to keep up with them. Don’t stress it, the fact is that a good amount of engineers are focused on being good with a few technologies than learning every new technology on the block. You don’t have to be an engineer to understand shifting trends in software engineering. Almost every time there is someone working on continuously improving how software is built, for example is gaining momentum as the new standard for building API’s, companies are adopting the use of for container orchestration, is the leading technology for containerization and service virtualization, the (Function as a service) ecosystem is offering companies a pay as you go model to scale their backend infrastructure and achieving a “ ” architecture . These are all very relevant trends that you should understand as a product manager and will help you communicate better with your team. GraphQL Kubernetes Docker FaaS serverless is a great resource with in-depth analysis to follow on new technology services and infrastructures from various networks of developer and communities. The New Stack Tweetzine from explaining HTTP 2 Mariko Kosaka 4. Be conversant with technology platforms offering backend as a service. if you are working directly with engineers in your day to day role. You should be familiar with technologies for real-time communication like , and application and API performance monitoring tools like , , , and , communication delivery tools like , and , Identity management tools for authorization like and , search and discovery tools like and and automation platform like and and deep linking platforms like . As a product manager, these are tools that you can introduce into your development workflow to help your team automate tasks that would take a very long time to spin up a backend for or even manage. Help your team focus on what matters instead of managing your own infrastructure especially if you are a startup with a small team. Firebase Pusher Socket.io, New Relic Datadog Apigee Cloud watch Sentry Sendgrid Twilio Message bird AuthO Okta Algolia Elasticsearch Zapier Airtable Branch You can follow and , both are community first resource sites where teams from top technology companies share their technology stack as well as advice on how to use them. Stackshare Siftery 5. API’s and how they deliver value for your business. If software is eating the world then API is eating software. API’s are basically the bread and butter of software. Chances are very high that your barber or hairstylist has an API they want you to plug into. If you are working on an API driven product it is rewarding that you understand the foundational terminology and have a mental model of what an API is. Everybody uses an API and it’s the current way to build abstraction in terms of the way business leverage technology to deliver value, maintain agility, and leverage its own data. So chances that you are not working on such product are slim. You already know what an API is, but you can still go further by learning how API’s are designed, what webhooks are, protocols for client-server interaction and authentication, how to bundle API into SDK’s, data formatting and what API gateways patterns are. Even how to test your API’s by firing up an HTTP Client to make a few requests to a server. An easy way to start using an API is with an HTTP Client like , an API development environment program that lets you quickly build HTTP requests to test with. You specify the URL, headers, and body, and the program sends it to the server properly formatted. Even if you decide not to try your hand at implementing an API, you can feel comfortable when they come up in conversations. is a good place to start learning what API’s are and how you can start querying API endpoints yourself. Another resource is and where you can learn the basics of API management. Postman Postman’s blog Zapier’s — Introduction to API course API academy . Here you can see the Postman’s solutions complete list 6. Learn continuous integration and continuous deployment best practices. if you are big on adopting the best agile practices in your company then you should know that continuous Integration, delivery, and deployment (CICD) are best workflows for great engineering culture and quality. In fact, you should be an advocate for such standards that drive automation and continuous improvement. Most important of all, these standards of shipping software product will lead to a more happier team which of course is what you want. By adopting such standards you will, in turn, drive a more collaborative, experimental and quality first approach to product development. The fact is your engineering team do not expect you to be the one to implement a CI/CD pipeline but they expect you to be forthcoming in your approach. Make the extra effort to have a basic understanding of CI/CD automation suite and how to get a continuous integration workflow set up. Be proactive in your approach, don’t wait for your ship to come out, swim to it. This introductory article from will get you started easily on how to approach continuous integration and delivery as a product manager and this guide from is an easy to read resource on CI/CD. You can make extra effort to check out ebooks from . Mind the Product Martin Fowler’s Oreilly books 7. Leverage data analytics and customer segmentation tools. Smaller and tighter-knit Startups are typically not built out to have a product manager for each role, in the case where bigger tech companies have a product manager for growth and SEO, customer experience, platform product managers and even for each product. You are probably interfacing with your sales and marketing team if there is any, to begin with. The experience of learning to multi-task, developing self-learning skills on the job, and being adaptable to all the several hats you wear every day can be exhausting. Resourcefulness is your biggest strength, what tools are you leveraging to help you overcome difficulties? With tools and platforms like , , , and , you can create experiments and scale A/B testing across your customer lifecycle to validate and measure the impact of new or redesigned features and get results on your dashboard within a short period of time. Before you would have to stretch your company’s resources and team to do this, but with just a simple setup and configuration you can start collecting and measuring user engagement within hours. The more you interact with these platforms and tools the more you become more familiar with them and can hack them to your specifications. Optimizely Segment Mix panel Kissmetrics 8. Querying and managing data. Storing, querying, and manipulating data is a skill set that almost every product manager posses but writing such SQL queries can be difficult and take a long time to write. You can leverage powerful database client like and with a friendly GUI to help you query and manipulate your database in no time. is a PostgreSQL desktop client and Tableplus is a tool for relational databases like MySQL, PostgreSQL, SQLite & more. You don’t need to have a strong command of SQL to run quick queries in your database. “ provides an easy to use interface, making Postgres more accessible for newcomers and specialists alike”. Postico Tableplus Postico Postico Conclusion I have mentioned a range of resources to learn from but by far the best community I have seen is by Ben Halpern, Jess Lee, and Peter Frank. Personally, I say it’s the best because it promotes a culture of inclusiveness and encourages healthy discussions and debates. The Practical Dev Also, register and attend engineering workshops and webinars. They are more hands-on than reading engineering blogs or forums. It’s easy for you to say “ its not your responsibility to figure the best technology to use or the software architecture of a product” but then again you are working in an industry that requires you to be a perfectionist and generalist. Saying “I don’t know” doesn’t have to be a bad thing, “I don’t know” gives someone the opportunity to help you. And it’s important that you work with people that promote a culture of togetherness and team-first approach to work. Cultivate a habit of curiosity. Learning new things is far easier and can be done much faster than everyone thinks. You are going to get stuck at something but be persistent with learning and whenever you can, ask for help from a teammate. Self-education is a lifelong curiosity. P.S. — If you need book recommendations, you can check out this list of top 10 software architecture resources on . O’Reilly’s online learning platform to get you started

Practices to help you become a technical product manager

Lessons I’ve learned from working with hundreds of developer SaaS businesses while building and managing developer-focused marketplaces at Heroku and Manifold Background During my time working on developer marketplaces, I’ve been exposed to hundreds of developer-focused businesses of all shapes and sizes, ranging from solo API operators working from their basements, to goliaths like IBM and their cloud division. I’ve had the opportunity to dive in deep on problems ranging from pricing optimization, converting tens of thousands of freemium users to paid accounts, co-sales agreements between large enterprise organizations, and the tangled maze of complexity that is known as HIPAA compliance. The one big thing that I’ve taken away from my experience is that the developer product market is still on the cusp of a Cambrian explosion, and there is nothing stopping you from being a part of it. In this post I’ll explain why the landscape is still good, if not great, for independent developer tools and services. I’ll also provide some tips for finding opportunities of your own and resources for getting started. Disclaimer: All opinions and findings are my own and not a reflection of either of my former companies. I’ve written this solely to share what I’ve learned over the years and because I believe there is potential for a lot more exciting growth in the industry. I hope that you find some value from my experience, and possibly get inspired to build your own developer tool or service. :) First, let’s bust a few myths “Developers don’t pay for products” You’re right. Developers and hobbyists may not pay for products, but teams and companies responsible for delivering business value in a constrained period of time pay for products. Put 100 developers in a room and 80 of these people will not pay for your product. But don’t miss the 20 folks in the back of the room willing to pay quite a lot to solve their business problems. definitely << The categories on ripienaar’s free-for-dev awesome list . There are well over 500 products on this list alone. “There are already too many developer tools and services” Smarter people than I have been calling for years. And there are to developers. The fact remains that year after year, the sheer number of businesses that exist online continues to grow, while the needs of existing businesses continues to grow as well. An increase in brick-and-mortar businesses means you need more plumbers, electricians, cleaning services, accountants, and in a similar vein the overall pie for developer products will continue to grow with the number and size of online businesses. the developer tools gold rush more than just a few services that are already available “(insert major cloud) is eating the world” AWS, GCP and Azure are all building their respective Death Stars, but in the same way that people are willing to deviate from Walmart, McDonald’s, and Starbucks, developers will always seek services that cater to them for various unique reasons, whether it’s specific use-cases, better documentation, or more knowledgeable support for their particular vertical. It’s unlikely, if not impossible, that one company will create the end-all product for all developers, use-cases, and verticals, just as there isn’t a single TV, phone, or laptop. Caveats For every there is a , for every a . Developer products are not a sure-fire path to success. But for the many of you out there with an entrepreneurial spirit and a knack for building useful tools, there is still a lot of land to grab in the developer tools space. Heroku Parse MongoDB RethinkDB The ever-growing importance of developer products In 2018 it is unthinkable for a developer to build an application without relying on at least one service. In reality, you’re likely using dozens of services, though they go by various names: PaaS, SaaS, IaaS, DBaaS, API, add-ons, among others. Take a look at to see the sheer number of developer products that the biggest tech companies use to run their businesses. StackShare Check out StackShare to see the services and tools that your favorite companies rely on to run their businesses The upside for developer product businesses continues to gain momentum In recent years we’ve seen a lot of activity from developer products pursuing IPOs and acquisitions. In terms of IPOs, the list of high profile developer product companies continues to grow with , , , , , , among others. The acquisitions list is even larger, with notable acquisitions like and . Many of these companies come from humble beginnings, with New Relic originating as a Ruby gem popular with Heroku users. SendGrid New Relic Twilio MongoDB Elastic Atlassian Treasure Data by Arm AppDynamics by Cisco Though most people focus on the headline grabbing exits, I’ve personally seen countless products go from two people in a garage to a few dozen people making a few million dollars in annual revenue in just a few years. A friend of mine took what he hoped best-case would bring in $1,000/month to supplement his income, but ended up turning into a $400,000+/year business within three years. Another team I worked with was working on a completely unrelated product at YC, then pivoted, taking an internal tool they were using and using it to transform into a fast-growing developer tools business with over $8M in funding to date. There are also countless individuals and teams who build healthy side projects, supplementing their incomes by building useful tools for fellow developers to generate a mostly passive income. Many of your favorite services are built and managed during nights and weekends as passion projects, or as side hustles from agency or other full-time work. More than one path Although most of my examples focus around profitability, there are alternative routes for building developer tools. One of the most prominent funding platforms is , which lets you create a community to transparently collect and spend funds, and developers have also been known to use tools like and to support their development. Open Collective Patreon Buy Me a Coffee The continued evolution of the space It’s important (though obvious) to note that the year 2018 is not the end-state for developers. History has shown us that Oracle will beget technologies like mongoDB, Redis, and Postgres, which will beget technologies like CockroachDB. There are a large number of powerhouse developer product companies today, but they will all be overturned by continuous innovation by developers like you. Finding your own pickaxe to sell There are tons of great resources already in regards to . In this section I’m going to focus on what I’ve encountered from my own experiences, specifically in regards to developer businesses. finding ideas for startups Looking within Start by looking at the tools you’ve built internally for your own needs. Odds are that if it’s useful to you, it can be useful to others. Many teams have taken services they were using internally and “productized” them for the greater world, only to realize that it was a bigger business than what they were originally pursuing. Marketplace “idea arbitrage” Find existing tools that are doing well on certain marketplaces, and offer the same type of service to other marketplaces. For example if you see a MongoDB product doing particularly well on Heroku, you can assume customers on Google Cloud may need a similar solution. Based on my experiences, you could take most of the services on AWS that aren’t already on marketplaces like Heroku and bridge the gap for their developers. Improving what isn’t “broken” I’ve seen countless startups succeed by taking a product they thought was inadequate and putting their own spin on it. Don’t like the APM you’re using? Build a better one. Think the API your business is relying on could be better? You’re probably not alone. Although we’re conditioned to think there isn’t honor in copying and improving, many successful developer products were a byproduct of frustration more than anything else. Much of the time, what feels “broken” may just need a twist or an angle that is specific to your use-case, that is likely shared by your future customers. Going developer friendly Riffing off of the last suggestion, there are a number of products that end up targeting the enterprise and leaving developers behind. Maybe the product becomes less developer friendly, the pricing becomes untenable, or they cut developer plans altogether. That doesn’t necessarily mean that it was unprofitable to serve developers, it’s likely they wanted to move upstream for bigger deals and to appease their shareholders. If you’re not interested in building the next enterprise developer company, this can be an opportunity for you to pick up a healthy base of disgruntled and abandoned developers. -aaS new technologies Often you will have developer product businesses focused on their open source business, . Often the technology may be pre-GA, or it’s a smaller team who can’t afford to split focus at the time. It can become a perfect opportunity to build a hosted service business. If you look at Heroku there are half a dozen Redis providers, all with successful businesses. If you look at MongoDB, built a business hosting MongoDB for years before the Atlas product launched, and they now co-exist. It’s arguable that mLab even has more love from developers than MongoDB’s own product. Even if a new technology is pre-GA, it’s likely that there are folks out there who are going to use the software in production, with or without your help. deferring the hosted business for a few years mLab Getting customers Now I’m admittedly biased given my marketplace background, but in my experience the easiest place to get started is marketplace integrations. How marketplaces work In exchange for a 15–30% revenue share, marketplaces on platforms like , , , etc will let you list your product as an “add-on” to their PaaS and IaaS products. In return you get exposure to their developer base, as well as opportunities for co-marketing and a smooth integration with their platforms. Heroku GCP AWS is a great starting point to get some early exposure and validate your idea with their beta program. , and after you have a dozen or so people test to make sure it’s working, you offer a free plan for Heroku users to beta test. In my experience Heroku customers are most prone to spend money on things like add-ons, as they’re both cultured into it and the nature of Heroku’s product attracts a type of person who prefers convenience. There are a number of businesses who do a large majority of their revenue on Heroku alone. Heroku You can set up a simple integration The next step from there would be to move into something like , which and has the benefit of being multi-cloud and aims to be a “Steam for Developers”. Manifold has very similar APIs You can also opt to join any of the classic Death Stars like , , or , though I’ve heard mixed reviews about their ability to drive leads to smaller independent shops. Google Cloud Microsoft Azure AWS Beyond that, there are lots of startups who have built healthy direct businesses using word of mouth, networking at conferences, or using various forms of paid social or newsletter ads. If you’re looking to get your feet wet and test an MVP though, I’d recommend starting with a marketplace. Resources to take you further There are tons of resources that can help you in your journey to build a profitable developer business. One of my favorite sources for high quality content is . Their focus tends to be high growth companies that want to scale to the enterprise, but there is a lot of good information that is generally applicable to all developer product businesses. consists of everything from to . They also have some if you’re local in the Bay Area, and many of them are free to attend. Heavybit Their library pricing strategies content strategy great events Heavybit has tons of great videos, blog posts, and podcasts that talk about all the intricacies of building and scaling developer products. Some really great communities that aren’t specific to developer products but have developer followings are and . Indiehackers is a great community of like-minded folks trying to build profitable online business and side projects. tends to target consumers, but when we were building Manifold we found a lot of success with Product Hunt’s community, and they were great for driving traffic and providing valuable feedback. Indiehackers Product Hunt Product Hunt Lastly, if you’re reading this you probably already know this but is a great resource for technically minded founders and has a wealth of information like this article on . Hacker Noon How to build a SaaS with $0 You absolutely don’t have to build a profitable developer SaaS business in 2018 Don’t get me wrong, it is incredibly difficult to build a product and business from nothing. If you’re happy with your current career or the product you’re building, you can stay the course. But if you’ve been looking for a new opportunity, want to try something new, or just need a side project to occupy your time, developer products are still a really exciting space. If you’ve been on the fence about building a developer product, hopefully I’ve piqued your curiosity! I hope you found some of what I had to share useful, and don’t hesitate to reach out in the comments, on , or if you’ve got questions or feedback! I know this was very high level since there was a lot to cover, so if you’re interested in any topic in particular, please let me know and I’m happy to write a follow-up post. Twitter via email

You can absolutely build a profitable developer SaaS business in 2018

Is a dev shop, agency, friend, or an offshore team building your product? If you answered yes, then this article may help you. I recently spoke with a few non-technical founders who were having their products built by others. I noticed a trend, and my advice to these founders was always the same, so I decided to write it down in hopes that it might help you. Here are the issues I noticed and the advice I gave. Own the code Some of the founders I spoke with didn’t know where their code was and had never seen it. One founder couldn’t even retrieve it from their last developer. ✔ It is important to own and control the code you’re paying for! I recommend you create a or a account to store your code. github.com bitbucket.org You will need to grant access to the developers so they can upload the code. write In Github I recommend you create an and . organization teams An organization is a collection of projects. A project is what holds your code. For a mobile app, you might have a project for the Android code, a project for the iOS code, and a project for the app server code. A team is a collection of users. A team (set of users) can be given access to your organization and all of its projects. Doing this makes it easy to add and remove developers. Another perk is that you can see all of the activity. The files that were added, removed, or changed and by whom. Own the accounts (app stores, SaaS integrations, etc…) Most of the founders I spoke with didn’t have access to one or more of their accounts. Your application will require many accounts. These might be app store accounts, software integration accounts, email accounts, etc… ✔ You should be the owner/admin of all your accounts. If you’re developing a mobile app, you should be the account owner for both the Apple and Android app stores. For both of these, you will be able to invite the developers and grant them access to your apps. Maybe you only want to grant developers the ability to upload new app versions but deny them the ability to publish them. You should be in control of who can access your app store account and who can modify it. You might have other accounts for things like 3rd party SaaS integrations. An example of this is Twilio. Let's say your application uses a Twilio phone number to send SMS messages. You will need to grant the developers access to your Twilio account so they can set up the integration. Owning all the accounts puts you in charge of who can access what. A good developer will ask you to create the accounts and invite them. Own the cloud Most of the founders I spoke with knew they were using the cloud but didn’t have the account information and didn’t know any of the details. If your application includes a website, API, or backend, then you are using cloud resources, like AWS, Google, Azure, Firebase, Heroku, etc... ✔ You should be the owner/admin of your cloud account. You should also understand what resources are being used an why. These resources have a monthly cost associated with them. For the first 12 months, it is usually free. You should ask your dev team what the monthly cost will be after the free period is over. You can grant developers read or write access to your cloud account. Know your stack Some of the founders I spoke with knew a little bit about their tech stack, but not many. Unfortunately one was using a, less than desirable stack. ✔ The application stack that is used to build your application is important. An application stack is the programming language, frameworks, libraries, tools, and cloud architecture, that is used when building an application. It would be bad if your application was built with an old, unpopular stack that not a lot of developers wanted to work on. It would be hard to find developers to work on it in the future. It would also be bad if your application was built with a totally new stack that hadn’t been adopted by the community yet. Some devs just want to work on the newest language/framework/library. The problem is that this new language/framework/library might become unpopular or abandoned. Before you pick a dev team ask them what stack they plan to use. Run this by a few developer friends of yours to see what they think. This is also a great resource to see what everyone else is using and what is popular, . https://stackshare.io Get deliverables None of the founders I spoke with were happy with the deliverables they have and were receiving, especially after I told them what was possible and in their best interest. ✔ It is important to get deliverables quickly and often. A deliverable can be anything from an architecture document, design mockups, skeleton app, testing framework, new features, bufixes, etc… Getting deliverables quickly and often is important because… 1.) It reduces your risk profile. If the developers you’ve hired are terrible you’ll know that sooner without spending all your money. Imagine if you only got one deliverable at the end of the project and that deliverable was awful. You would have spent all your dough for nothing. 2.) It forces your developers to implement a continuous deployment approach early, which is needed if you want to iterate quickly… which you do. The longer the time between when you ask your developer to implement something and when you actually see it the more money you will spend. This is because communication isn’t always 100% and things sometimes seem different when you actually use it. Summary The most important thing when building your product is to get to market quickly and with as little resources 💸 as possible in order to validate your product market fit. It is also important to own everything and be able to switch developers when needed. This list is by no means complete, it is just a list of the most common issues I found when advising non-technical founders. Please feel free to comment if you know of other major pain points that plague non-technical founders going it their own. Thanks for reading my article. I hope it helped. I know how much heart and energy goes into developing a product. Please don’t hesitate to reach out if you need help or just want to chat. I love helping founders.

Checklist for non-technical founders

Winds started out as a simple example app for Stream, but thanks to an outpouring of support from our awesome community we decided to focus more time and energy on the project. The initial response around Winds 2.0 has exceeded all of our expectations. Since launching in mid-May the application ranked #1 on Hacker News for over a day, has 5,500 stars (and counting) on Github and became a trending app on Product Hunt. Going into it, there was no way to anticipate how popular Winds 2.0 would become — would it be an absolute hit or an epic failure and waste of time?? The team enjoys building example apps so I knew it wouldn’t be a total loss, and it’s been rewarding to see this new iteration being used so extensively in the first month after release. The tech stack for Winds is completely different from that of Stream. You may have seen the blogpost that StackShare how Stream powers the activity feeds for 300 million users using Go, RocksDB and Raft. Winds, however, is based on , , , , , and . wrote about Node.js MongoDB Atlas Express PM2 Bull Babel React To get started with Winds 2.0, you try the web version or download the application , or if you feel more adventurous head over to and spin it up locally. Next up, let’s talk a little bit about the Winds 2.0 stack and why we chose to go with the technologies we did (and why we chose to build Winds in the first place!). here GitHub RSS is a Broken Experience 😭 We realize that many RSS power users are developers, designers and journalists. One of our goals with Winds is to answer the questions we had been asking ourselves: What if a community of developers and designers could create an RSS experience that’s simplistic and polished? Could this reverse the downward spiral of less users taking advantage of the technology and more publications dropping support? The future of RSS is uncertain at best. Our hope with this project is to make a contribution to #ReviveRSS. Why JavaScript/Node…? 🤔 Another core goal for Winds is to enable a wide range of developers to contribute. We want it to be easy for to be able to notice something they don’t like about their RSS/Podcast experience and easily submit a pull request with a fix or addition. anyone If you were brave enough to explore the codebase, you probably noticed that we’re using JavaScript for everything — both front and backend. Most of our team is experienced with Go and Python, so Node was not an obvious choice for this example app. What’s funny about JavaScript is how many people complain about it being an inadequate language. Sure, it has its quirks — single threaded, callback hell, etc.– but we believe that it’s possible to build great software in any language. For Winds, JavaScript has been a great choice to foster a community around the project. More importantly, JavaScript’s maturity has started to shine with the added support of the syntax. Async/Await Sure… there will be haters who refuse to acknowledge that there is anything remotely positive about JavaScript (there are even rants on about Node.js.); however, without writing completely in JavaScript, would not have seen the results we did. Here’s a quick breakdown of some of the reasons why chose JavaScript: Hacker News we we Nearly every developer knows or can, at the very least, read JavaScript With and Node.js v10.x.x, it’s become a very capable language ES6 / is powerful and easy to use ( ) Async Await Async/Await vs Promises allows us to experiment with next-generation JavaScript (features that are not in the official JavaScript spec yet) Babel allows us to consistently install packages quickly (and is filled with ) Yarn tons of new tricks DevOps 🖥️ It’s rare that you hear about how a company deploys and manages code. Being that Winds 2.0 is open-source, we wanted to share a few of the tools we use to get the job done when it comes to getting our code from our machines up to the server. The web version of Winds is statically hosted on S3 with CloudFront. In all, it costs us a few dollars a month to host. Every desktop version of the application is deployed inside of Electron, allowing us to bridge the gap between web and desktop. As for the back-end API, that’s another story. We have a rather decent deploy flow going on to ensure stability and maintainability. Here’s the rundown: All code is stored on GitHub We manually kick off builds on AWS using a combination of and Fabric Boto create a fresh Winds environment consisting of instances, (ASG), (ELB), and a instance CloudFormation EC2 Auto Scaling Groups Application Load Balancer Redis AWS CCM stores and retrieves the various configurations required at boot (e.g. current version, etc.) Dotenv & Environment variables are stored in Puppet and CCM Once all EC2 instances are available, a script runs and applies the configuration on all live instances (in apply mode) Puppet boots, automatically starting the various Node.js processes we need to keep our application alive ( and ) PM2 API Workers For logging metrics, we use a combination of + + . StatsD Graphite Grafana Understanding Electron ⚡ We wanted to experiment with building an app with downloads for every Linux distro, macOS, and Windows, in addition to the web. Fundamentally, this seemed pretty easy: we write code, wrap it in an Electron shell, and release to our desired operating system… It turns out we were wrong. Electron Electron, though powerful, turned out to be a bigger beast than we had anticipated. Building to different distros was especially hard, even with (granted, we had the bad luck of needing to patch electron-builder (and that bug has since been fixed), but that only accounted for some of the pain points we hit). The macOS menu bar had to be just right for the macOS store to accept our app, and performing small tasks with the Electron API, like opening a link in an external browser, turned out to be pretty difficult. Despite the difficulties, our team moved forward with some custom tooling (all visible and open-sourced on ) and we released not only to all of our release targets but to the , too. electron-builder GitHub web Testing in JavaScript 🗳️ JavaScript is still the wild west to a degree. It’s rather un-opinionated, especially if you’re using Express, so we had to roll our own testing framework to get the job done. Our API, which is built with Express, uses a combination of various Node.js modules. Here’s a list of the tools we use for testing: as a testing framework Mocha as an assertion library Chai as our mocking library Sinon as the HTTP mocking library Nock as a module mocking library mock-require as our test coverage tool Istanbul Here’s an actual example of our . Bonus: test runner The combination of test modules we chose our team to move fast with multiple developers working on various feature sets at the same time, without bringing down the API. Frontend 🛠️ React is a phenomenal framework, and in our opinion, has won the battle against other frameworks such as Angular and Ember. Given its updated , it’s perfect for the Winds 2.0 project. MIT license The main stack that we use for Winds 2.0 is pretty straightforward: : Main Stack Create-react-app React (react-redux) Redux React-router Electron Now let’s chat about some of the front-end modules we used to make Winds 2.0 a reality. : Interesting Modules is a nice React interface to the core audio element API React-audio-player for scrolling events to handle automatic pagination React-waypoint for easy OPML file imports React-dropzone for flawless image fallbacks in the event we don’t have an image stored React-image Backend When you’re building a large application, you generally rely on many libraries and tools to increase the code quality, time to market, etc. With that said, Winds too relies on many libraries and tools. Below are several, but not all, that we use: FeedParser is a rather complex Node.js module that in our opinion, is the backbone of the project. It handles most of the inconsistencies found in RSS feeds and spits out a “cleansed” version of the feed. Without this module, we would be writing a lot of if/else statements… and that’s no fun. FeedParser Franc-Min is a language detection module that we utilize for determining the language of a feed. This might sound like a small task; however, it’s in fact, a large part of our personalization engine. For example, we only recommend feeds in English to users who speak English. Same with other languages. Franc-Min Bull helps keep the Winds 2.0 queue structurally sound with the help of . It comes with a super easy API and supports multiple queues, which is a perfect fit for our use-case. Additionally, there are several open-source monitoring tools on their page that provide insight into what is happening behind the scenes. Bull Redis GitHub ES6 enables our team to write minimalist code for a wide range of people. All of the code in Winds 2.0 is 100% JavaScript (with the exception of a handful of bash scripts to help with deploy workflows). The team is currently migrating much of the functionality in the codebase to utilize to reduce the number of lines of code. JavaScript w/ ES6 Async/Await Yarn is absolutely amazing. It’s an incredibly fast package manager built specifically for JavaScript. On top of that, it’s 100% open-source and nearly always available, due to it’s caching mechanisms. We’ve used in the past, and although it works just fine, the team here at Stream prefers Yarn. Yarn npm Axios is a Promise based HTTP client for the browser and Node.js. We actually use it on both the front and back-end for various tasks. For example, all front-end HTTP requests flow through a wrapper to Axios. And for the back-end, we utilize Axios to inspect the size of the file prior to sending them through the parsing process and then off to the database for storage — this ensures large files don’t bring down our worker processes. If you haven’t checked out Axios, you definitely should. Axios Commander is another Node.js module, this time providing full support for building command-line interfaces. Yes, that’s right, Winds has a CLI that we use for various tasks such as testing feeds, dropping RSS feeds, and more! Commander Babel “allows us to use the next generation of JavaScript, today”. Essentially, if a feature, such as imports isn’t available in a particular JavaScript (front-end and/or back-end), we can still use it by leveraging Babel. Babel Express is used to power our . Compared to other frameworks out there, it truly shines when under stress. Honestly speaking, our team has experience with most, if not all, of the JavaScript frameworks and we find Express to be the easiest to work with. It’s regularly maintained, has open-source components, and it’s awesome. 😎 Express API Sentry allows for real-time crash reporting for our back- and front-end. What blows us away is how granular you can get with Sentry. Their features help us identify and debug errors and give us insight on when to fix or rollback. When it comes to firefighting, this tool definitely wins the market. Sentry Algolia provides lightning-fast ( ) search for our application. In 2ms, our users can discover RSS feeds and podcasts to read or listen to. They even have components for frameworks such as React and to make the integration better. This is by far one of our favorite; however, we like them all. 😛 Algolia literally under Angular Stream is a key resource to Winds 2.0, as it provides news feeds and activity streams for our users and even machine learning-based personalization. Without , we would not be able to serve up suggested content to our users as we currently do. Stream Stream MongoDB Atlas is a phenomenal DBaaS, allowing us to worry about acquiring users, while MongoDB worries about uptime. It’s identical to hosting your own cluster, except MongoDB provides you with a dashboard and a URI to connect to. With MongoDB Atlas, there’s no more worrying about cluster health, monitoring, etc. MongoDB Atlas Mongoose is a powerful ODM that allows us to define rich models within our MongoDB environment. Generally speaking, are schemaless (meaning they have or require no form); however, with MongoDB, it’s always a good idea to specify a schema so you can index and organize your data properly. This allows for easy updates and efficient queries. Mongoose NoSQL databases PM2 is a Node.js process manager. It allows us to ensure uptime, and scale processes as we need. The CLI for the project is dead simple, which allowed our team to pick it up on the fly. PM2 Final Thoughts 😁 To get started with Winds 2.0, you can try the web version or download the application . If you’re feeling more adventurous head over to and spin it up locally. here GitHub RSS is in a vicious circle. Winds is a community effort to help turn the tide and . Contributions are always much appreciated. To discuss new features check out the . #ReviveRSS official Slack channel for Winds If you’re curious to know a little more about and how our API works, we have an easy that will walk you through the process of building scalable activity feeds. Stream 5-minute API tour

The Winds 2.0 Stack

Founder Interview Disclosure: , the API for scalable feeds, has previously sponsored Hacker Noon. Stream Today, we’re going to catch up with their CEO, Thierry Schellenbach about the origin, scale, and direction of his company. David : Let’s start with some numbers. What’s the scale and traction of your company? : and I started Stream about 3 years ago. At the start, it was just the two of us doing all engineering work. Soon after, we were accepted into Techstars NYC — which is when things really took off for us. As of today, we have 600 companies relying on . We also power the feeds of over 300 million end users and handle approximately 34 billion feed updates a month. It’s hard to believe that we’ve had this type of growth in such a small amount of time! Thierry Tommaso Stream 300 million end users. That’s a lot. In terms of distribution, are your larger customers at the top taking up most of that? Yes, several of our largest customers have user counts in the hundreds of millions. Of course, not all of those are active, but our large customers drive a significant portion of that 300 million number. Cool. So you really are an expert in news feeds. They’ve really blown up in the last 10–15 years — going from something that didn’t really exist to something that supplies much of our news. If you could, please give me a walkthrough of the technological milestones that made the news feeds we expect possible, and what upcoming breakthroughs for the news feed that you are excited about. As you know, companies like Yahoo, Facebook, Twitter and Linkedin have invested heavily on their feed technology. Back in early 2010, in order to keep up with their massive growth they had to push the limits of what was available at the time. Many of us still remember Facebook’s blackouts in the early days or Twitter’s fail whale. One huge breakthrough has been the creation of NoSQL databases. For scaling news feeds it’s necessary to have a very capable and high throughput database. Operating news feed at scale requires a very capable and high throughput database. Projects like Cassandra, Kafka, RocksDB, Twemproxy, FlockDB, Voldemort and Tornado were created by companies working on some of the largest news feeds and are the building blocks of many of the major news feeds we see today. It’s also interesting to note how only 15% of our customers build social apps — which is what people initially think of when they think about feeds. Feeds these days power content discovery in a wide range of applications. Developers are leveraging Stream for B2B applications, transport systems, educational platforms, music, sports, e-commerce, etc. And so do you think that NoSQL has reduced the amount of time to load stories and find the right stories to load? Definitely, Stream for instance uses a combination of Go, RocksDB, and Raft. ( Read this StackShare post to learn about the technology that enables Stream to power the feed for over 300 million users .) What are the common reasons companies plug into your API instead of building a news feed on their own? The biggest reason we see companies integrating with Stream is time to market. Another big reason is cost. Try to think about building your own in-house feed solution and all the moving parts and components associated with that — it’s a daunting thought! Many feeds, for example, run around 9 different clusters of servers that consist of the following: API servers Message brokers Autoscaling worker clusters Database clusters (Cassandra is a common option) Real-time infrastructure Redis as a backend for distributing real-time work & handling locks Analytics infrastructure Machine learning APIs Machine learning worker infrastructure It’s a headache to build, document, monitor and maintain. Even the most simple feeds can become expensive as your user base grows. Typically, Stream is substantially cheaper than hosting an in-house solution. We’ve even seen companies that had already created a feed in-house, switch to Stream simply because it’s more affordable. As it so happens, this was a big reason why we created Stream. I had this exact experience at my previous company. We needed to build feed technology in-house and it quickly became a very expensive piece of our technology stack. This is a common theme we see. Take companies like Instagram, even when they were small, they were spending huge amounts of money on just hosting and maintaining their feeds. also talks about their in-house vs Stream experience in a case study of ours. Dubsmash I think you talked a little bit about this earlier, but what aspect of Stream is driving down the cost of hosting as a competitive advantage for you? Our entire team is focused on building feed technology for over 600 companies. Because of this, we have to spend a lot of time and effort on high availability, monitoring, and optimizations. Part of the cost advantage comes from this focus and our ability to spread the work over 600 apps. On the tech side, the cost advantages comes from 3 capabilities of Stream: We use an in-house database called Keevo, built on top of & , which is optimized specifically for feeds (Linkedin and Instagram do something similar). RocksDB Raft Stream uses a combination of fanout on-read and fanout on-write. Our various services are powered by Go (except and machine learning, which use Python). www.getstream.io Those are the technical aspects that stand out most, but there’s also hundreds of smaller optimizations that have compounded over the last few years. And what’s the simplest way to see the power of your API? As with many things, I believe that when you’re a developer and you’re trying out a new solution, it really helps if it’s easy to use. With that in mind, we created a 5 minute interactive tutorial: . It allows you to try out Stream’s API right there in your browser. getstream.io/try-the-api/ Could you tell us about your previous company, Fashiolista? We started that company around nine years ago now. It was a bit like Pinterest before there was Pinterest. We went through a period of crazy growth. Where were your users coming from? Our users mostly came from magazines. We had a few large magazines and bloggers. Influencers started pushing the app and sharing their profiles, which drove a ton of traffic. If you take a look at popular fashion bloggers, you’ll see they have a huge audience. That space is pretty big. We grew to a few million members in just a few months, which was an exciting experience. However, once we had millions of members, everything seemed to fall apart. While we were able to easily fix most of the things that broke, the one thing we kept having difficulties with was the feed. Where would it break and why would it break? Feeds are particularly tricky to scale because the data is highly connected. It’s not a linear problem, it becomes substantially harder as the size of your network grows. The first thing you’ll notice is that the feed will start to load slowly for some users, which is a very common problem. Twitter used to have their . In the beginning, Facebook had five second loading time. The issue stems from all data being connected to each other. Fail Whale So backing up, I read that you also created an open source solution for building feeds. How was that received by the community? I started about five years ago. At the time there were some good papers from but no open source solutions. Stream-Framework quickly became the most widely used open source solution for building feeds. Stream-Framework Yahoo & Princeton about feed technology Because of this I ended up talking to many developers that were developing feeds. While Stream-Framework was substantially easier than building it from scratch, it was still a huge hassle to set up, maintain, and build an API around. The minimum cost of deploying a scalable and highly available solution was also very high. That was one of the reasons for building Stream. Stream’s tech stack is highly optimized and easy to access via an API. It can be very hard to go from an open-source project to a company. So what advice do you have for others that are trying to do the same? And when I say hard, I mean from the idea of logistically doing it. My advice would be to take note of industry trends and give the people what they want. The most relevant and interesting trend we noticed was that developers were starting to leverage APIs for everything that’s not unique to their app. Companies started providing specific components of an app as a service. Think Sendgrid for email, Algolia for search, Twilio for SMS, Stripe for payments, Mapbox for maps. The list goes on and on. Stream is that lego piece for adding feeds to your application. I personally think this is a great time for developers to start a company. If you have deep domain expertise around a certain topic you can leverage that to build a company. It’s of course hard, but it’s becoming more and more of a possibility. So backing up to your customers. You’re in an interesting spot because you’re seeing news feed activity across different industries and across different use cases. I was curious about the iterations on the news feed for your customers. So how are you seeing them learn and evolve from what Stream did to how they want Stream to give a better experience for their customers, and how that kind of feedback loop has been working? How do you see that working? One feature I’m seeing quite often lately is . For instance, say a friend of yours opens up an app and adds a hundred items, the app developers/designers will have those updates aggregate depending on whatever makes sense for their platform. Aggregation is very helpful for reducing noise in the feed. aggregation Another thing I see more and more apps doing is a “while you were away” type of experience. Take Facebook and Twitter for example. While there are a few of these platforms where users come back to daily, the users of majority of apps using feeds come back once a week, or even once a month. You better win that moment, when they come back, with good content. Exactly. We see not only see apps creating aggregated feeds, but also using personalization in addition to the basic chronological version of their feed. This way users get tailored, relevant content when they return to the app. While those are the main things we see, of course real-time is still a big part of it as well. Many developers will make sure their feed updates in real-time. A few years ago, this used to be a very trendy topic. Nowadays it’s taken for granted — users expect things to update while you’re looking at them. So what does Stream look like down the road, in say five or 10 years? I mean, what is the long-term vision? I think the most interesting part about feeds is that they are the leading way through which we discover content online. Whether your following people on Instagram, RSS on Feedly or , topics on Quora, music on Spotify, creators on Youtube or stocks in financial apps, the underlying tech is based on feeds. Apps nowadays still struggle with feeds, since the technology is relatively young. It’s exciting to see how Stream can help these apps improve content discovery. Winds Within the onboarding experience, how do you make sure you show the power of Stream and move people through quickly? I guess that’s an interesting project in and of itself. As developers we’ve researched and tested MANY products, so we know what really works and what doesn’t from experience. Once we created a version of ‘ ’ that we thought was powerful, we then asked our developer friends to go through the tutorial while we sat next to them. That qualitative feedback was super helpful to make Stream easy to use. Try The API

How Stream Started Building News Feeds

Greetings from the other side of the Internet, passed away yesterday. He was 70. His obituary by ’s Cindy Cohen can be read . Mr. Barlow co-founded EFF and it promotes freedom of expression in digital media. If you have 3 minutes, read (excerpt below) while listening to (a song he wrote). John Perry Barlow Electronic Frontier Found here A Declaration of the Independence of Cyberspace Grateful Dead’s Cassidy “Cyberspace consists of transactions, relationships, and thought itself, arrayed like a standing wave in the web of our communications. Ours is a world that is both everywhere and nowhere, but it is not where bodies live. We are creating a world that all may enter without privilege or prejudice accorded by race, economic power, military force, or station of birth. We are creating a world where anyone, anywhere may express his or her beliefs, no matter how singular, without fear of being coerced into silence or conformity.” A deep breath for the internet’s free thinkers. Onto our top 22 tech stories of the last week: Dev Stories by . “I should say at the outset that there’s no real point to this blog post. I had a thing to do, I did the thing, and was moderately pleased with the result. I enjoy reading of such tales from other developers, so I thought I’d share my story. Recommended music pairing for this post: the new Awolnation album.” Writing an easing function; a slightly interesting story David Gilbertson by Why you should stop using product roadmaps and try GIST Planning Itamar Gilad by . 20 new features, a redesign and a rewrite. Boom! Announcing CodeSandbox 2.5 Ives van Hoorne by . The 4th annual StackShare Awards! 🎉 Top 50 Developer Tools of 2017 StackShare by . Oh, inside the challenges of building a do-to list! Interview with Simon Chan on the Future of Microsoft To-Do John Gruber by . “How do you control people from talking too much? If they do talk way too much that the allocated time, there are some subtle and fun ways to let them know they have gone over the time, bring in a timer, squeaky toys, or mini football to pass around once their time is up.” You are doing Stand-ups wrong! Juliet Lara Blockchain by . “There are a few blockchain supply chain use cases which have the potential to restructure the industry. One particularly impactful use case is making it easier for small suppliers to get more financing on better terms.” A Multi-Trillion Dollar Blockchain Use Case Taylor Pearson by . “Despite the fact of the non existence of an Ethereum Wallet or Browser for React Native to interact with cross platform decentralized mobile apps, it’s possible bring the blockchain to React Native.” Bringing the blockchain to React Native locropulenton by . “While it’s the undisputed business hub of the Middle East, Dubai has set an ambitious goal: To be the first blockchain-powered government in the world by 2020.” Can an Entire City Run on the Blockchain? Quora by . “Most of your ownership records are stored in paper ledgers. These can be tampered. The data on Blockchain cannot be altered. On a Blockchain there are two things. A block and a chain. At a very high level, it is just a chain of blocks.” Popular Use Cases of Blockchain Technology You Need to Know Febin John James Bitcoin by . “The objective of this article is to examine the fundamental path to a price $100,000 per bitcoin…” Bitcoin’s Substantive & Technical Road to $100K Kerati Apilak by . “Mining Ethereum blocks takes about 14 seconds per confirmation compared to Bitcoin’s 10 minutes.” The Flippening: Ethereum vs. Bitcoin Kenny Li More Crypto World by . “A trading system is about discipline. And most people won’t put in the effort to make it work for them. They won’t go through all the steps necessary to become a master.” My Super Secret Crypto Turtle Strategy Revealed Daniel Jeffries by . “Despite a down month for the market at large in January, cryptocurrencies are still rapidly growing as technological assets and consumer products.” 4 Cryptocurrency Trends to Watch in 2018 TechRecaps by . Curious about where IOTA’s headed? Read this. IOTA: Real-world use cases are coming Chris Mueller (aka IOTASUPPORTER) by . “Last year ICOs raised $5 billion across 800+ projects and even more are expected this year. There is a lot of enthusiasm but still more questions than answers. One of the pressing questions is how ICOs will be regulated by the SEC. There are no firm directives yet but there are growing legal standards on what to consider before launching an ICO.” Think like the SEC before you ICO Brittany Laughlin by . “Jay Clayton, Chairman of the SEC, and Christopher Giancarlo, Chairman of the CFTC, testified in front of Congress on cryptocurrencies. Here are the 10 most important takeaways from Jay Clayton’s remarks.” Top 10 Points Made By the SEC and CFTC Congress Testimony On Cryptocurrencies Howard Marks Oh, Facebook… by . “I’ve been following Facebook’s run since it’s birth in 2004. I was running at the time, and we were the largest social network in the world for a brief time in between Friendster & MySpace. I was buying websites started by 16, 18, 20 year old kids, and I was the first person to call Mark and offer to buy the company, when he had about 1,000 kids at Harvard on it.” Facebook’s EPIC Run Comes To An End Lou Kerner Bolt Media Marketing by . “It’s like saying: . It is true: on average, American presidents are, and were, much taller than the average American. Yet though being tall is an advantage, height wasn’t the core driver behind people who later became presidents. It took a lot of talent, resilience and luck to get to the presidency.” Don’t Start with Why! Yonatan Kagansky in order to be an American president, you need to start with being relatively tall Startup Talk by . “Plenty of potential $100 million companies fail because they raised capital as if they would become billion dollar unicorns. The problem is that it’s hard to tell how big an idea can become at the outset. It wasn’t immediately obvious that an operating system for Altair microcomputers, a search algorithm for academic papers, or a social network for Harvard students would become three of the defining tech companies of our times!” Are You Working on a Side Project or a Startup? Founder Collective by . **“**The purpose of your story should do two things: (1) Identify who you are as cofounders and that you are able to run a company together without dysfunction. (2) You have the ability to build what you want to build, and, if your idea doesn’t work out, the ability to build something else.” How to Secure Your First Round of Funding Will Goto … And Don’t Forget About Physics by . “It’s not the act of seeing that changes the circumstances of particles, it’s the other variables that must be present in order for us Heisenberg’s Uncertainty Principle Doesn’t Mean You Have Telekinesis Alex the Younger to see.” Until next time, don’t take the realities for granted. Kind Regards , David Smooke @AMI

What to Read in Tech

is an API that enables developers to build news feeds and activity streams ( ). We are used by over 500 companies and power the feeds of more than 300 million end users. Companies such as Product Hunt, , Powerschool, , , Compass and Fabric (Google) rely on Stream to power their news feeds. In addition to the API, the founders of Stream also wrote the most widely used . Stream try the API Under Armour Bandsintown Dubsmash open source solution for building scalable feeds Here’s what Stream looks like today: Number of servers: 180 Feed updates per month: 34 billion Average API response time: 12ms Average real-time response time: 2ms Regions: 4 (US-East, EU-West, Tokyo and Singapore) 1B API requests per month (~20k/minute) Given that most of our customers are engineers, we often talk about our stack. Here’s a high level overview: is our primary programming language. Go We use a custom solution built on top of + for our primary database (we started out on but wanted more control over performance). stores configs, API keys, etc. RocksDB Raft Cassandra PostgreSQL and handle tracing, and provide detailed monitoring and we use the stack for centralized logging. OpenTracing Jaeger StatsD Grafana ELK is our language of choice for machine learning, devops and our website ( , & ). Python https://getstream.io Django DRF React Stream uses a combination of and . This results in fast read performance when a users open their feed, as well as fast propagation times when a famous user posts an update. fanout-on-write fanout-on-read and I ( ) are the developers who started Stream nearly 3 years ago. We founded the company in Amsterdam, participated in Techstars NYC 2015 and opened up our Boulder, Colorado office in 2016. It’s been quite a crazy ride in a fairly short amount of time! With over 15 developers and a host of supporting roles, including sales and marketing, the team feels enormous compared to the early days. Tommaso Barguli @tschellenbach The Challenge: News Feeds & Activity Streams The nature of follow relationships makes it hard to scale feeds. Most of you will remember Facebook’s long load times, Twitter’s fail whale or Tumblr’s year of technical debt. Feeds are hard to scale since there is no clear way to shard the data. Follow relationships connect everyone to everyone else. This makes it difficult to split data across multiple machines. If you want to learn more about this problem, check out these papers: Twitter’s tech back in the days with 150 million active users LinkedIn’s ranked feeds Yahoo/Princeton research paper At a very high level there are 3 different ways to scale your feeds: Fanout-on-write: basically precompute everything. It’s expensive, but its easy to shard the data. Fanout-on-read: hard to scale, but more affordable and new activities show up faster. Combination of the above two approaches: better performance and reduced latency, but increased code complexity. Stream uses a combination of fanout-on-write and fanout-on-read. This allows us to effectively support both customers with highly connected graphs, as well as customers with a more sparse dataset. This is important since the ways in which our customers use Stream are very different. Have a look at these screenshots from , Unsplash, and : Bandsintown Product Hunt Switching from to Python Go After years of optimizing our existing feed technology we decided to make a larger leap with 2.0 of Stream. While the first iteration of Stream was powered by Python and Cassandra, for Stream 2.0 of our infrastructure we . The main reason why we switched from Python to Go is performance. Certain features of Stream such as aggregation, ranking and serialization were very difficult to speed up using Python. switched to Go We’ve been using Go since March 2017 and it’s been a great experience so far. Go has greatly increased the productivity of our development team. Not only has it improved the speed at which we develop, it’s also 30x faster for many components of Stream. The performance of Go greatly influenced our architecture in a positive way. With Python we often found ourselves delegating logic to the database layer purely for performance reasons. The high performance of Go gave us more flexibility in terms of architecture. This led to a huge simplification of our infrastructure and a dramatic improvement of latency. For instance, we saw a 10 to 1 reduction in web-server count thanks to the lower memory and CPU usage for the same number of requests. Initially we struggled a bit with package management for Go. However, using together with the contributed to creating a great workflow. Dep VG package If you’ve never tried Go, you’ll want to try this online tour: https://tour.golang.org/welcome/1 Go as a language is heavily focused on performance. The built-in tool is amazing for finding performance issues. Uber’s library is great for visualizing data from PPROF and will be bundled in PPROF in Go 1.10. PPROF Go-Torch Switching from Cassandra to & Raft RocksDB 1.0 of Stream leveraged for storing the feed. Cassandra is a common choice for building feeds. Instagram, for instance started, out with but eventually to handle their rapid usage growth. Cassandra can handle write heavy workloads very efficiently. Cassandra Redis switched to Cassandra Cassandra is a great tool that allows you to scale write capacity simply by adding more nodes, though it is also very complex. This complexity made it hard to diagnose performance fluctuations. Even though we had years of experience with running Cassandra, it still felt like a bit of a black box. When building Stream 2.0 we decided to go for a different approach and build . Keevo is our in-house key-value store built upon RocksDB, gRPC and Raft. Keevo is a highly performant embeddable database library developed and maintained by Facebook’s data engineering team. RocksDB started as a fork of Google’s LevelDB that introduced several performance improvements for SSD. Nowadays RocksDB is a project on its own and is under active development. It is written in C++ and it’s fast. Have a look at how this . In terms of technology it’s much more simple than Cassandra. This translates into reduced maintenance overhead, improved performance and, most importantly, more consistent performance. It’s interesting to note that also uses RocksDB for . RocksDB benchmark handles 7 million QPS LinkedIn their feed Our infrastructure is hosted on and is designed to survive entire availability zone outages. Unlike Cassandra, Keevo clusters organizes nodes into leaders and followers . When a leader (master) node becomes unavailable the other nodes in the same deployment will start an election and pick a new leader. Electing a new leader is a fast operation and barely impacts live traffic. AWS To do this, Keevo implements the Raft consensus algorithm using . This ensures that every bit stored in Keevo is stored on 3 different servers and operations are always consistent. This site does a great job of visualizing how Raft works: Hashicorp’s Go implementation https://raft.github.io/ Not Quite Microservices By leveraging Go and RocksDB we’re able to achieve great feed performance. The average response time is around 12ms. The architecture lies somewhere between a monolith and a microservice. Stream runs on the following 7 services: Stream API Keevo Real time & Firehose Analytics Personalization & Machine learning Site & Dashboard Async Workers To see all our services divided into stacks, head over . here Personalization & Machine Learning Almost all large apps with feeds use machine learning and personalization. For instance, LinkedIn prioritizes the items in your feed. Instagram’s explore feed displays pictures outside of the people you follow that you might be interested in. Etsy uses a similar approach to optimize ecommerce conversion. Stream supports the following 5 use cases for personalization: Documentation for building personalized feeds. All of these personalization use cases rely on combining feeds with analytics and machine learning. For the machine learning side we generate the models using . The models are different for each of our enterprise customers. Typically we’ll use one of these amazing libraries: Python — LightFM How to implement recommender systems XGBoost Scikit learn , and Dask Numpy Pandas (For development) Jupyter Notebook Mesa Framework Analytics Analytics data is collected using a tiny Go-based server. In the background, it will spawn go-routines to rollup the data as needed. The resulting metrics are stored in . In the past, we looked at , which seems like a solid project. For now, we could get away with a simpler solution though. Elastic Druid Dashboard & Site The dashboard is powered by and . We also use React and Redux for all of our example applications: React Redux Cabin: A tutorial series on building a social network with React & Redux (Winds 2.0 is coming out very soon) Winds: Open source RSS & Podcast app The site, as well as the API for the site, is powered by , and . Stream is sponsoring Django Rest Framework since it’s a pretty great open source project. If you need to build an API quickly there is no better tool than DRF and Python. Python Django Django Rest Framework We use to resize the images on our site. For us, Imgix is cost efficient, fast and overall a great service. is a good open source alternative. Imgix Thumbor Real time Our real time infrastructure is based on Go, Redis and the excellent library. It implements the Bayeux protocol. In terms of architecture it’s very similar to the node based library. gorilla websocket Faye It was interesting to read the “ ” post on Hacker News. The author moves from Go to Node to improve performance. We actually did the exact opposite and moved from Node to Go for our real time system. The new Go-based infrastructure handles 8x the traffic per node. Ditching Go for Node.js Devops, Testing & Multiple Regions In terms of devops the provisioning and configuration of instances is fully automated using a combination of: CloudFormation Cloud-Init Puppet Boto & Fabric Because our infrastructure is defined in code it has become trivial to launch new regions. We heavily use . Every single piece of our stack is defined in a CloudFormation template. If needed we are able to spawn a new dedicated shard in a few minutes. In addition, AWS Parameter Store is used to hold application settings. Our largest deployment is in US-East, but we also have regions in Tokyo, Singapore and Dublin. CloudFormation A combination of Puppet and Cloud-init is used to configure our instances. We run our self-contained Go’s binaries directly on the EC2 instance without any additional containerization layer. Releasing new versions of our services is done by . Travis first runs our test suite. Once it passes, it publishes a new release binary to . Common tasks such as installing dependencies for the Go project, or building a binary are automated using plain old Makefiles. (We know, crazy old school, right?) Our binaries are compressed using UPX. Travis GitHub Tool highlight: Travis has come a long way over the past years. I used to prefer in some cases since it was easier to debug broken builds. With the addition of the aptly named “debug build” button, Travis is now the clear winner. It’s easy to use and free for open source, with no need to maintain anything. Travis Jenkins Next we use to do a rolling deploy to our AWS instances. If anything goes wrong during the deploy it will halt the deploy. We take stability very seriously: Fabric A high level of test coverage is required. Releases are created by Travis (making it hard to deploy without running tests). Code is reviewed by at least 2 team members. Our extensive QA integration test suite evaluates if all 7 components still work. We’ve written about our experience with . When things do break we do our best to be transparent about the ongoing issue: testing our Go codebase StatusPage for 24/7 phone support Twilio for notifying our team VictorOps #firefighting channel on Slack for customer support Intercom VictorOps is a recent addition to our support stack. It’s made it very easy to collaborate on ongoing issues. Tool Highlight: VictorOps The best part about is how they use a timeline to collaborate amongst team members. VictorOps is an elegant way to keep our team in the loop about outages. It also integrates well with . This setup enables us to quickly react to any problems that make it into production, work together and resolve them faster. VictorOps Slack The vast majority of our infrastructure runs on AWS: for load balancing ELB for reliable Postgres hosting RDS for Redis hosting ElastiCache for our DNS Route53 for our CDN Cloudfront for ELK and tracing with AWS ElasticSearch Jaeger The devops responsibilities are shared across our team. While we do have one dedicated devops engineer, all our developers have to understand and own the entire workflow. Monitoring Stream uses for tracing and for beautiful dashboards. The tracking for Grafana is done using . The end result is this beauty: OpenTracing Grafana StatsD We track our errors in and use the ELK stack to centralize our logs. Sentry Tool Highlight: OpenTracing + Jaegar One new addition to the stack is . In the past we used , which works like a charm for Python, but isn’t able to automatically measure tracing information for Go. OpenTracing with is a great solution that works very well for Stream. It also has, perhaps, the best logo for a tracing solution: OpenTracing New Relic Jaeger Closing Thoughts is an absolutely amazing language and has been a major win in terms of performance and developer productivity. For tracing we use OpenTracing and Jaeger. Our monitoring is running on StatsD and Graphite. Centralized logging is handled by the ELK stack. Go Stream’s main database is a custom solution built on top of and Raft. In the past we used Cassandra, which we found hard to maintain and which didn’t give us enough control over performance when compared to RocksDB. RocksDB We leverage external tools and solutions for everything that’s not a core competence. hosting is handled by ElastiCache, Postgres by RDS, email by , test builds by Travis and error reporting by Sentry. Redis Mailgun Thank you for reading about our stack! If you’re a user of Stream, please be sure to add to your stack on StackShare. If you’re a talented individual, ! And finally, if you haven’t tried out Stream yet, take a look at this quick . Stream come work with us tutorial for the API This post was originally written by Thierry Schellenbach, CEO at GetStream.io . The original post can be found at https://stackshare.io/stream/stream-and-go-news-feeds-for-over-300-million-end-users .

Stories, News & Mentions about STACKSHARE

Stackshare’s Stories on HackerNoon