Automagical Architecture

A breakdown of a modern startup’s tech stack with a focus on JavaScript and simplicity.

Overview

Automagical is a platform that makes it easy and fun to create quality marketing videos from existing content.

Disclaimer: I’m the founder & CTO of Automagical.

I’d like to give an overview of its architecture and reflect on how important engineering simplicity is to a product’s success.

Automagical makes heavy use of JavaScript, as the backend is written in Node.js, and the frontend is built with React.js via the create-react-app standard. The source code is managed by git as a monorepo on GitHub, and deployment is handled by the extremely simple now service.

The primary backend datastore is a hosted version of mongoDB, and two hosted instances of redis are used for storing user authentication tokens and async task information respectively. An async task queue is used to manage long-running operations like video rendering and video trimming.

Google Cloud Platform is used for long-term storage (similar to AWS S3) and to host the video renderer component which has some special requirements.

A large part of how we’ve been able to scale Instagram with very few engineers is by choosing simple, easy-to-understand solutions that we trust. (source)

Services

The platform is broken up into the following logical services. Each service can be scaled horizontally based on usage.

Backend API

This service provides the core backend API powering the Automagical platform (https://api.automagical.ai). It is a Koa webserver that exposes a REST API which is consumed by the frontend webapps. It is deployed with a single command via now.

Backend Queue

This service provides an async task queue worker for rendering and processing other long-running tasks. It is the only service hosted on Google Cloud Platform instead of now because the renderer module requires more compute resources.

Frontend Webapp

This service provides the core Automagical video webapp (https://app.automagical.ai) built with create-react-app where users create, edit, and render storyboards. It is deployed with a single command via now.

Frontend Marketing Site

This service provides the Automagical marketing site (https://automagical.ai) built with create-react-app. It is deployed with a single command via now.

Modules

The platform also includes the following modules which encapsulate well-defined, core functionality.

Renderer Module

This module is used by the backend queue task runner. It is a 2D renderer for converting Automagical storyboards into videos. It is built using Node.js, headless-opengl, and FFmpeg.

Summarize Module

This module is used by the backend API service to generate summaries from html. It uses a variety of metrics, NLP, and machine learning techniques to generate quality extractive text summaries from source articles.

DevOps

Once deployed, Automagical is designed to run with minimal maintenance. In the unlikely event of a server crash, the affected service automatically restarts and an error is logged with Google Cloud Error Reporting.

All environment secrets like Stripe access keys, database passwords, etc. are handled by local .env files which are deployed along with the code via now. The only thing a new developer has to do to be able to run and deploy the Automagical platform is gain access to the git monorepo and the accompanying private .env files.

Monitoring

• Every API call is logged with Google Cloud Trace.
• Uptime is used to send emails and SMS alerts in the unlikely event of a service being unavailable.
• Google Analytics is setup to monitor both the marketing site and main webapp, so you can view both live and historical traffic data.
• All major webapp actions (like creating a storyboard, adding a new scene, rendering a video, etc.) are sent to both Google Analytics and Intercom, so you can track the path a specific user takes through the application.

Takeaways

• Node.js is an excellent programming language for iterating quickly, but you still need to take time up front to understand and sketch out which parts of your system make sense to isolate into separate services or modules.
• Microservices are trendy and great once you need to scale, but 99% of early stage startups don’t need them. Start simple with an eye towards logical, monolithic services in your app, and only consider adding the complexity of solutions like microservices and container orchestration once you need to.
• Using Zeit now for deployment has been hugely successful in allowing us to focus our efforts on building the core product and ignore most problems surrounding deployment / scaling / orchestration. It also has such a simple API that there’s virtually zero vendor lock-in, so once we need to scale, we can assess the best option and change course quickly.
• Our engineering burn rate is extremely lean at around $500 per month, with the majority of that budget coming from third-party API vendors. When considering building a feature versus paying for an equivalent third-party API, lean towards using SaaS APIs as much as possible if your budget allows, so you can focus on building out your core product. If this cost becomes a hindrance down the line, then build a custom version yourself only once you need to.
• Fleshing out and executing on an MVP is very, very difficult. If there are any doubts that a feature or polish item should be in an MVP, skip it and validate your product with potential customers first before spending extra time developing. Early adopters care more about the product idea and its future potential than specific features, and their feedback will be invaluable in prioritizing your product roadmap. This sounds easy & obvious, but in practice, it takes a lot of experience and constant vigilance to maintain this mindset.
• Until you’re confident you’ve found product / market fit, focus on keeping your engineering efforts as simple and lightweight as possible. The ability to quickly iterate on product changes in the early days of a startup is significantly more important than other technical concerns. This tradeoff between development velocity and maintainability takes experience to optimize for, and even the best engineers tend to overdo it early on with too much structure and process. KISS.

Everything in engineering is a tradeoff && KISS rules all.

Summary

Automagical is a modern Node.js platform architected with care and a focus on simplicity. It’s broken up into several main services: backend API, backend queue, frontend webapp, and frontend marketing site. The core IP is broken into well-encapsulated components, namely the renderer module and the summarize module. Zeit’s extremely simple now service is used for one-command deployments, and all databases (mongodb and redis) are hosted by third-parties for ease of maintenance.

For more details or specific questions, feel free to reach us at [email protected].

Before you go…

If you liked this article, click the 👏 below, and share it with others so they can enjoy it as well.