The Road So Far part 1 : overview part 2 : testing and continuous delivery strategies part 3 : ops part 4 : building a scalable push notifications system part 5 : building a better recommendation system Since quite a few people have asked about the serverless architecture we ended up with and some of the things we have learnt along the way. Yubl’s closure As such, this is the first of a series of posts where I’d share some of the lessons we learnt. However, bear in mind the pace of change in this particular space so some of the challenges/problems we encountered might have been solved by the time you read this. ps. many aspects of this series is already covered in a talk I gave on at in 2016, you can find the slides and recording of the talk . Amazon Lambda Leetspeak here From A Monolithic Beginning Back when I joined Yubl in April I inherited a monolithic backend running on EC2 instances, with (hosted ) and (hosted ) thrown into the mix. Node.js MongoLab MongoDB CloudAMQP RabbitMQ There were numerous problems with this legacy system, some could be rectified with incremental changes (eg. blue-green deployment) but others required a rethink at an architectural level. Although things look really simple on paper (at the architecture diagram level), all the complexities are hidden inside each of these 3 services and boy, there were complexities! My first tasks were to work with the ops team to improve the existing deployment pipeline and to draw up a list of characteristics we’d want from our architecture: able to do small, incremental deployments deployments should be fast, and requires no downtime no lock-step deployments features can be deployed independently features are loosely coupled through messages minimise cost for unused resources minimise ops effort From here we decided on a architecture, and seemed the perfect tool for the job given the workloads we had: service-oriented Amazon Lambda lots of APIs, all HTTPS, no ultra-low latency requirement lots of background tasks, many of which has soft-realtime requirement (eg. distributing post to follower’s timeline) To a Serverless End It’s suffice to say that we knew the migration was going to be a long road with many challenges along the way, and we wanted to do it incrementally and gradually increase the speed of delivery as we go. “The lead time to someone saying thank you is the only reputation metric that matters” – Dan North The first step of the migration was to make the legacy systems publish state changes in the system (eg. user joined, user A followed user B, etc.) so that we can start building new features on top of the legacy systems. To do this, we updated the legacy systems to publish events to streams. Kinesis Our general strategy is: build new features on top of these events, which usually have their own data stores (eg. etc.) together with background processing pipelines and APIs DynamoDB , CloudSearch , S3 , BigQuery , extract existing features/concepts from the legacy system into services that will run side-by-side these new services will initially be backed by the same shared database MongoLab other services (including the legacy ones) are updated to use hand-crafted API clients to access the encapsulated resources via the new APIs rather than hitting the shared database directly MongoLab once all access to these resources are done via the new APIs, data migration (usually to tables) will commence behind the scenes DynamoDB wherever possible, requests to existing API endpoints are forwarded to the new APIs so that we don’t have to wait for the iOS and Android apps to be updated (which can take weeks) and can start reaping the benefits earlier After 6 months of hard work, my team of 6 backend engineers (including myself) have drastically transformed our backend infrastructure. Amazon was very impressed by the work we were doing with Lambda and in the process of writing up a case study of our work when Yubl was shut down at the whim of our major shareholder. Here’s an almost complete picture of the architecture we ended up with (some details are omitted for brevity and clarity). Some interesting stats: 170 Lambda functions running in production roughly 1GB of total deployment package size (after cleans up unreferenced versions) Janitor Lambda Lambda cost was around 5% of what we pay for EC2 for a comparable amount of compute the no. of production deployments increased from 9/month in April to 155 in September For the rest of the series I’ll drill down into specific features, how we utilised various AWS services, and how we tackled the challenges of: centralised logging centralised configuration management distributed tracing with correlation IDs for Lambda functions keeping Lambda functions warm to avoid coldstart penalty auto-scaling AWS resources that do not scale dynamically automatically clean up old Lambda function versions securing sensitive data (eg. mongodb connection string, service credentials, etc.) Links Slides and Recording of my Amazon Lambda talk at Leetspeak Janitor-Lambda function to clean up old deployment packages Like what you’re reading but want more help? I’m happy to offer my services as an and help you with your serverless project — architecture reviews, code reviews, building proof-of-concepts, or offer advice on leading practices and tools. independent consultant I’m based in and currently the only UK-based . I have nearly of with running production workloads in AWS at scale. I operate predominantly in the UK but I’m open to travelling for engagements that are longer than a week. To see how we might be able to work together, tell me more about the problems you are trying to solve . London, UK AWS Serverless Hero 10 years experience here I can also run an to help you get with your serverless architecture. You can find out more about the two-day workshop , which takes you from the basics of AWS Lambda all the way through to common operational patterns for log aggregation, distribution tracing and security best practices. in-house workshops production-ready here If you prefer to study at your own pace, then you can also find all the same content of the workshop as a I have produced for Manning. We will cover topics including: video course authentication authorization with API Gateway Cognito & & testing running functions locally & CI/CD log aggregation monitoring best practices distributed tracing with X-Ray tracking correlation IDs performance cost optimization & error handling config management canary deployment VPC security leading practices for Lambda, Kinesis, and API Gateway You can also get the face price with the code . Hur­ry though, this dis­count is only avail­able while we’re in Manning’s Ear­ly Access Pro­gram (MEAP). 40% off ytcui

Amazon

Yubl’s road to Serverless — Part 2, Testing and CI/CD

Ask me for help about serverless

Read My Stories

Too Long; Didn't Read

Make resilience your competitive advantage

Yubl’s road to Serverless — Part 1, Overview

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

6 Tips To Scale an AppSync Project To 200+ Resolvers That Will Blow Your Mind

101 Stories To Learn About Cloud Infrastructure

10 Things in Engineering We Don't Spend Enough Time On

10 Things I Did To Increase CloudTrail Logs Security

10 reasons to give cloud computing a go

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps