Businesses can’t afford any downtime, especially when users demand speed and constant availability. But what happens when the database supporting your application gets choked from your application’s traffic? This was the situation we encountered while scaling a system with over a billion user records. Therefore, this article describes how we migrated a production database with no downtime, keeping users logged in, transactions flowing, and business running as usual. But before jumping into how we did it, it’s worth asking, why does zero-downtime even matter so much in today’s architecture? Why Zero-Downtime Matters More Than Ever In today’s web application development, particularly in SaaS or consumer apps, downtime translates to lost revenue, broken trust, and SLA violations. When I refer to “zero-downtime,” I mean a migration process where: User-facing endpoints remain fully accessible. No incomplete transactions. No broken sessions or corrupted data. No corrupted data. Building systems to support hundreds of thousands of concurrent users makes it absolutely clear that simple “scheduled maintenance” is a risk you can’t afford. With that goal in mind, let me walk you through the challenge we faced—and how we planned to overcome it. The Problem: Monolithic DB Under Pressure. We were using a monolithic setup of Postgres with read replicas; however, over time, the schema became a bottleneck. An increasing number of sessions would require a write, which would need to be followed by analytic queries and cron jobs, which would put IOPS through the roof. The two goals we faced were: Transition to a more horizontally scalable system, in this case, distributed Postgres. Positive transition with no downtimes or performance impacts. The solution required a phased migration strategy, starting with isolating reads and writes to give us control over database access. Step 1: Introducing a read and write proxy layer. The very first thing we did was create a proxy interface around our database calls. This is very similar to creating a small-scale ORM with awareness of reading and writing. All write requests were marked and routed to the main database, and reads were handled by the replicas. What this did was to allow us to have precise control during the initial stages of migration since we could reroute operations easily. Now, with a clean and solid abstraction layer in code is extremely helpful at this point. Unmanaged and scattered queries increase the amount of work for this single step for weeks. Once we had control over read-write traffic, the next step we took was to keep both systems in sync, without risking data integrity. Step 2: Dual writes with safety nets. Our approach was simple; we started by implementing dual writes. For some of our higher traffic models, we did dual writes on both the old and new databases. However, this approach can be risky. What happens if one of the writes fails? In our case, we added in place a logging mechanism that flagged where writes failed. We kept a log of all the failures and put all the discrepancies in a queue. These discrepancies could be resolved in the background without holding up the main process. I ensured that every dual write function had an idempotent built in. This ensured that even when someone is executing the same function multiple times, it had no negative impact. This made re-tries safer and the outcome expected. Dual writes keep the new system updated in real time, and we turned to the heavier lift, migrating the backlog of historical data. Step 3: Asynchronous Data Backfill Copying a billion records in one go is impossible, at least not without breaking something. Think of a database crossing a river by stepping on stones, wherein each stone is a 1000-record chunk, and you need to mark the stone as migrated to step safely. That is the approach that was taken by us by setting up a worker queue to handle 1000 record chunks and marking each “migrated” to fully make database usage as efficient as possible. To avoid “hitting” the database with too much traffic, we combined Kafka and batch processing. A “warmed up” database helped us to focus on active users as a priority. That way, the most valuable and important records are “fetched” in the most efficient manner. When the new database warmed up and got tested, we began the careful process of shifting live traffic, gradually and safely. Step 4: Feature Flags for Safe Cutover We had an over 95% success rate of our writes in the new system, and reads showed parity. We enabled a feature flag to switch to the new database for a small portion of traffic. We did this using LaunchDarkly. As our confidence increased, we extended the rollout to 100%. If you haven't started using feature flags for infrastructure changes, this is your sign. It changes everything from a constant gamble to a methodical approach. Switching over is just one half of the process. The other half is verifying that it worked and being prepared for what might break. Step 5: Post-Migration Verification Our job wasn’t done until we verified the following: Snapshot comparisons between old and new DBs, Query performance benchmarks, Fallback support in case we needed to roll back. We also left read-only access to the old system live for two weeks, just in case we needed to run forensic checks. After everything was in place, we reflected on what made this migration successful and what we’d do differently next time. Lessons Learned: Start with abstraction. Your migration is only as smooth as your system’s modularity. Test for reality. Load test every read, write, and edge cas,e not just happy paths. Keep observability high. Logs, metrics, tracing this isn't optional when migrating live systems. Design for humans. Developers fear migrations because they’ve been burned before. Build tooling that makes it safe and explainable. Final Thoughts: These takeaways proved essential, but the broader lesson was that relocating a billion-user database is not a casual weekend task; it is a milestone in engineering achievement. However, it is completely achievable with the proper tools, frameworks, and attitude, all while preserving the user experience. My two cents after training thousands of developers at Sumit’s platform is that zero downtime is not a marketing term; it is a genuine phrase based on concern for the users, team, and the developers themselves.

This story contains AI-generated text. The author has used AI either for research, to generate outlines, or write the text itself. 

Walkthroughs, tutorials, guides, and tips. This story will teach you how to do something new or how to do something better.

How We Migrated a Billion-Record Database With Zero Downtime

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

The Tradeoff Triangle: Why Microservices Start to Hurt at Scale

6 Database Migration Tools For Complete Data Integrity & More

A Comprehensive Guide for Backing Up and Restoring Azure SQL Databases

A/B Testing was a Jerk, Until we Found the Replacement for Druid

Balancing Simplicity and Auditability: Database Models for User Funds

Breaking Up a Monolithic Database with Kong

The Tradeoff Triangle: Why Microservices Start to Hurt at Scale

6 Database Migration Tools For Complete Data Integrity & More

A Comprehensive Guide for Backing Up and Restoring Azure SQL Databases

A/B Testing was a Jerk, Until we Found the Replacement for Druid

Balancing Simplicity and Auditability: Database Models for User Funds

Breaking Up a Monolithic Database with Kong

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps