Automagical Architecture

A breakdown of a modern startup’s tech stack with a focus on JavaScript and simplicity. Overview is a platform that makes it easy and fun to create quality marketing videos from existing content. Automagical Disclaimer: I’m the founder & CTO of Automagical. I’d like to give an overview of its architecture and reflect on how important is to a product’s success. engineering simplicity Automagical makes heavy use of JavaScript, as the backend is written in , and the frontend is built with via the standard. The source code is managed by git as a monorepo on , and deployment is handled by the extremely simple service. Node.js React.js create-react-app GitHub now The primary backend datastore is a hosted version of , and two hosted instances of are used for storing user authentication tokens and async task information respectively. An is used to manage long-running operations like video rendering and video trimming. mongoDB redis async task queue is used for long-term storage (similar to AWS S3) and to host the video renderer component which has some special requirements. Google Cloud Platform A large part of how we’ve been able to scale Instagram with very few engineers is by choosing simple, easy-to-understand solutions that we trust. ( ) source Services The platform is broken up into the following logical services. Each service can be scaled horizontally based on usage. Backend API This service provides the core backend API powering the Automagical platform ( ). It is a webserver that exposes a REST API which is consumed by the frontend webapps. It is deployed with a single command via . https://api.automagical.ai Koa now Backend Queue This service provides an async task worker for rendering and processing other long-running tasks. It is the only service hosted on instead of because the renderer module requires more compute resources. queue Google Cloud Platform now Frontend Webapp This service provides the core Automagical video webapp ( ) built with where users create, edit, and render storyboards. It is deployed with a single command via . https://app.automagical.ai create-react-app now Frontend Marketing Site This service provides the Automagical marketing site ( ) built with . It is deployed with a single command via . https://automagical.ai create-react-app now Modules The platform also includes the following modules which encapsulate well-defined, core functionality. Renderer Module This module is used by the backend queue task runner. It is a 2D renderer for converting Automagical storyboards into videos. It is built using Node.js, , and . headless-opengl FFmpeg Summarize Module This module is used by the backend API service to generate summaries from html. It uses a variety of metrics, NLP, and machine learning techniques to generate quality extractive text summaries from source articles. DevOps Once deployed, Automagical is designed to run with minimal maintenance. In the unlikely event of a server crash, the affected service automatically restarts and an error is logged with . Google Cloud Error Reporting All environment secrets like Stripe access keys, database passwords, etc. are handled by local files which are deployed along with the code via . The only thing a new developer has to do to be able to run and deploy the Automagical platform is gain access to the git monorepo and the accompanying private .env files. .env now Monitoring Every API call is logged with . Google Cloud Trace is used to send emails and SMS alerts in the unlikely event of a service being unavailable. Uptime is setup to monitor both the marketing site and main webapp, so you can view both live and historical traffic data. Google Analytics All major webapp actions (like creating a storyboard, adding a new scene, rendering a video, etc.) are sent to both Google Analytics and , so you can track the path a specific user takes through the application. Intercom Takeaways Node.js is an excellent programming language for iterating quickly, but you still need to take time up front to understand and sketch out which parts of your system make sense to isolate into separate services or modules. Microservices are trendy and great once you need to scale, but 99% of early stage startups don’t need them. Start simple with an eye towards logical, monolithic services in your app, and only consider adding the complexity of solutions like microservices and container orchestration once you need to. Using for deployment has been hugely successful in allowing us to focus our efforts on building the core product and ignore most problems surrounding deployment / scaling / orchestration. It also has such a simple API that there’s virtually zero vendor lock-in, so once we need to scale, we can assess the best option and change course quickly. Zeit now Our engineering burn rate is extremely lean at around $500 per month, with the majority of that budget coming from third-party API vendors. When considering building a feature versus paying for an equivalent third-party API, lean towards using SaaS APIs as much as possible if your budget allows, so you can focus on building out your core product. If this cost becomes a hindrance down the line, then build a custom version yourself . only once you need to Fleshing out and executing on an is very, very difficult. If there are any doubts that a feature or polish item should be in an MVP, skip it and validate your product with potential customers first before spending extra time developing. Early adopters care more about the product idea and its future potential than specific features, and their feedback will be invaluable in prioritizing your product roadmap. This sounds easy & obvious, but in practice, it takes a lot of experience and constant vigilance to maintain this mindset. MVP Until you’re confident you’ve found product / market fit, focus on keeping your engineering efforts as simple and lightweight as possible. The ability to quickly iterate on product changes in the early days of a startup is significantly more important than other technical concerns. This tradeoff between development velocity and maintainability takes experience to optimize for, and even the best engineers tend to overdo it early on with too much structure and process. . KISS Everything in engineering is a tradeoff && KISS rules all. Summary Automagical is a modern Node.js platform architected with care and a focus on simplicity. It’s broken up into several main services: backend API, backend queue, frontend webapp, and frontend marketing site. The core IP is broken into well-encapsulated components, namely the renderer module and the summarize module. Zeit’s extremely simple service is used for one-command deployments, and all databases (mongodb and redis) are hosted by third-parties for ease of maintenance. now For more details or specific questions, feel free to reach us at . dev@automagical.ai Before you go… If you liked this article, click the 👏 below, and share it with others so they can enjoy it as well.