Engineering Resilience: A Deep Dive into Chaos Engineering in Distributed Systems

Hyper-scalars with their inherent nature of distributed networks/systems, have completely changed the outlook of the word “software reliability”. With distributed systems, we can no longer rely just on traditional QA; but start to deal with the chaos that is inherent in communication with multiple systems. Chaos Engineering is the practice of intentionally experimenting on a system, based on our assumptions or understanding, and gaining understanding of the unknown behaviour of our systems, so we can gain confidence in its ability to withstand turbulent conditions in production. Chaos Engineering is not a random discipline, but a highly disciplined, systematic approach to finding vulnerabilities before they turn into 3 AM outages. As our architectures shift from predictable monoliths to micro-services built on hundreds of interdependent components spread over multiple cloud providers and SaaS solutions, the ability to catch failure modes through standard unit or integration testing has hit its limit. To understand this, let's start by understanding the mental model behind it. The Mental Model: Hypothesis Testing at Scale To practice Chaos Engineering in the right sense, engineering teams need to fundamentally shift how they think about complex systems. The New mental model goes likes this - Distributed systems are inherently chaotic and that failure isn't just possible, it's inevitable. Systems need to embrace failure as a natural occurrence. - Amazon CTO Werner Vogels Systems need to embrace failure as a natural occurrence. - Amazon CTO Werner Vogels Challenging the Fallacies of Distributed Computing Challenging the Fallacies of Distributed Computing The reason why we need Chaos Engineering comes down to "Fallacies of Distributed Computing” where a classic set of dangerous assumptions are made, which leads to fragile architectures. When developers assume that the network is reliable, latency is zero, or bandwidth is infinite, they build systems unprepared for production. Chaos experiments are designed to break these assumptions and understand where and when these assumptions break. Assumptions of Distributed Computing Production Reality Chaos Experiment Countermeasure The network is always reliable Stuff breaks. Cables get cut, routers choke, and packets just drop in transit. Dropping about 5% of packets or forcing a hard network partition. Latency is zero Physics gets in the way. Physical distance and crowded pipes mean data takes time to travel. Slapping a 500ms to 2-second delay on traffic to see what times out or breaks. Bandwidth is infinite A sudden, unexpected traffic spike will easily choke our app. Flooding the network to max out capacity and watching how the system handles the bottleneck. Topology never changes Pods spin up, instances die, and IP addresses shift constantly. Killing random pods, containers, or EC2 instances without any warning. Transport cost is zero Moving data across regions actually costs real money and adds noticeable lag. Cutting off a whole Availability Zone or region to see if fail-overs actually work. Assumptions of Distributed Computing Production Reality Chaos Experiment Countermeasure The network is always reliable Stuff breaks. Cables get cut, routers choke, and packets just drop in transit. Dropping about 5% of packets or forcing a hard network partition. Latency is zero Physics gets in the way. Physical distance and crowded pipes mean data takes time to travel. Slapping a 500ms to 2-second delay on traffic to see what times out or breaks. Bandwidth is infinite A sudden, unexpected traffic spike will easily choke our app. Flooding the network to max out capacity and watching how the system handles the bottleneck. Topology never changes Pods spin up, instances die, and IP addresses shift constantly. Killing random pods, containers, or EC2 instances without any warning. Transport cost is zero Moving data across regions actually costs real money and adds noticeable lag. Cutting off a whole Availability Zone or region to see if fail-overs actually work. Assumptions of Distributed Computing Production Reality Chaos Experiment Countermeasure Assumptions of Distributed Computing Assumptions of Distributed Computing Assumptions of Distributed Computing Production Reality Production Reality Production Reality Chaos Experiment Countermeasure Chaos Experiment Countermeasure Chaos Experiment Countermeasure The network is always reliable Stuff breaks. Cables get cut, routers choke, and packets just drop in transit. Dropping about 5% of packets or forcing a hard network partition. The network is always reliable The network is always reliable The network is always reliable Stuff breaks. Cables get cut, routers choke, and packets just drop in transit. Stuff breaks. Cables get cut, routers choke, and packets just drop in transit. Dropping about 5% of packets or forcing a hard network partition. Dropping about 5% of packets or forcing a hard network partition. Latency is zero Physics gets in the way. Physical distance and crowded pipes mean data takes time to travel. Slapping a 500ms to 2-second delay on traffic to see what times out or breaks. Latency is zero Latency is zero Latency is zero Physics gets in the way. Physical distance and crowded pipes mean data takes time to travel. Physics gets in the way. Physical distance and crowded pipes mean data takes time to travel. Slapping a 500ms to 2-second delay on traffic to see what times out or breaks. Slapping a 500ms to 2-second delay on traffic to see what times out or breaks. Bandwidth is infinite A sudden, unexpected traffic spike will easily choke our app. Flooding the network to max out capacity and watching how the system handles the bottleneck. Bandwidth is infinite Bandwidth is infinite Bandwidth is infinite A sudden, unexpected traffic spike will easily choke our app. A sudden, unexpected traffic spike will easily choke our app. Flooding the network to max out capacity and watching how the system handles the bottleneck. Flooding the network to max out capacity and watching how the system handles the bottleneck. Topology never changes Pods spin up, instances die, and IP addresses shift constantly. Killing random pods, containers, or EC2 instances without any warning. Topology never changes Topology never changes Topology never changes Pods spin up, instances die, and IP addresses shift constantly. Pods spin up, instances die, and IP addresses shift constantly. Killing random pods, containers, or EC2 instances without any warning. Killing random pods, containers, or EC2 instances without any warning. Transport cost is zero Moving data across regions actually costs real money and adds noticeable lag. Cutting off a whole Availability Zone or region to see if fail-overs actually work. Transport cost is zero Transport cost is zero Transport cost is zero Moving data across regions actually costs real money and adds noticeable lag. Moving data across regions actually costs real money and adds noticeable lag. Cutting off a whole Availability Zone or region to see if fail-overs actually work. Cutting off a whole Availability Zone or region to see if fail-overs actually work. Moving from Unknowns to Knowns Moving from Unknowns to Knowns It helps us to think of our system as a matrix of known-unknowns. Traditional testing covers the "known knowns"—the behaviours we expect and understand. Chaos Engineering targets the "unknown unknowns." There are bizarre, cascading failures and complex component interactions that nobody anticipates until the system falls over in production. By scientifically testing our assumptions, we are dragging these vulnerabilities out of the dark and into the light, building the organisation's infrastructure's immune system. The Scientific Method to Chaos We can't randomly just unplug/shutdown servers and call it Chaos Engineering. The practice relies on four core principles that ensure experiments yield actual insights without impacting the business. 1. Define Your Steady State 1. Define Your Steady State First, we need to define baseline by answering ‘What does "normal" functioning look like?’. Define steady state business-level metrics that reflect the user experience, instead of focusing on CPU or memory usage. Steady State Metric Type Examples Why do we need this Business Output Orders processed per second, new signups, or video stream starts. If these numbers dip, the business is actively losing cash, and customers are probably complaining ( if not now, then in a few minutes/hours). User Experience P99 latency spikes, error rates, and Time-To-First-Byte (TTFB). We need to start looking at the outside system as a giant black box (for front-end, that would be back-end and vice-versa). Systemic Boundary Raw throughput or successful HTTP 200 requests per second. We need to start looking at the outside system as a giant black box (for the frontend, that would be the backend and vice versa). Steady State Metric Type Examples Why do we need this Business Output Orders processed per second, new signups, or video stream starts. If these numbers dip, the business is actively losing cash, and customers are probably complaining ( if not now, then in a few minutes/hours). User Experience P99 latency spikes, error rates, and Time-To-First-Byte (TTFB). We need to start looking at the outside system as a giant black box (for front-end, that would be back-end and vice-versa). Systemic Boundary Raw throughput or successful HTTP 200 requests per second. We need to start looking at the outside system as a giant black box (for the frontend, that would be the backend and vice versa). Steady State Metric Type Examples Why do we need this Steady State Metric Type Steady State Metric Type Steady State Metric Type Examples Examples Examples Why do we need this Why do we need this Why do we need this Business Output Orders processed per second, new signups, or video stream starts. If these numbers dip, the business is actively losing cash, and customers are probably complaining ( if not now, then in a few minutes/hours). Business Output Business Output Business Output Orders processed per second, new signups, or video stream starts. Orders processed per second, new signups, or video stream starts. If these numbers dip, the business is actively losing cash, and customers are probably complaining ( if not now, then in a few minutes/hours). If these numbers dip, the business is actively losing cash, and customers are probably complaining ( if not now, then in a few minutes/hours). User Experience P99 latency spikes, error rates, and Time-To-First-Byte (TTFB). We need to start looking at the outside system as a giant black box (for front-end, that would be back-end and vice-versa). User Experience User Experience User Experience P99 latency spikes, error rates, and Time-To-First-Byte (TTFB). P99 latency spikes, error rates, and Time-To-First-Byte (TTFB). We need to start looking at the outside system as a giant black box (for front-end, that would be back-end and vice-versa). We need to start looking at the outside system as a giant black box (for front-end, that would be back-end and vice-versa). Systemic Boundary Raw throughput or successful HTTP 200 requests per second. We need to start looking at the outside system as a giant black box (for the frontend, that would be the backend and vice versa). Systemic Boundary Systemic Boundary Systemic Boundary Raw throughput or successful HTTP 200 requests per second. Raw throughput or successful HTTP 200 requests per second. We need to start looking at the outside system as a giant black box (for the frontend, that would be the backend and vice versa). We need to start looking at the outside system as a giant black box (for the frontend, that would be the backend and vice versa). Once we've got a solid baseline, it's time to set up a hypothesis that we can actually measure and prove wrong. We can't just assume saying that "the system will handle it”; we need to quantify it. Example Scenario: "If I nuke a container, the load balancer needs to catch it and reroute traffic in under 30 seconds. Meanwhile, our P99 latency shouldn't spike past 250ms, and the error rate needs to stay completely under 0.1%." If the system fails any of those specific checks, our assumption was wrong and we got a bottleneck to fix. Example Scenario: 2. Simulate Real-World Disruptions 2. Simulate Real-World Disruptions Create chaos variables which will reflect production reality. Prioritise events based on their likelihood or potential blast radius—think in terms of server crashes, traffic spikes, or malformed API responses. 3. Run It in Production 3. Run It in Production Staging environments simply don't have the scale, messy data, or complex traffic patterns of the live system. Production is the only place where we can validate the authentic request path. 4. Minimize the Blast Radius 4. Minimize the Blast Radius While running chaos experiments in production is important, protecting the customer is the ultimate priority. Start small, target a single container or a tiny fraction of traffic. Widen the scope only once the system can handle a subset of experiments. Controlled Chaos Techniques Implementation Strategy For what Benefit Targeted Blast Radius Instead of hitting the main cluster, we target a single canary group or just one non-critical microservice. It stops a controlled burn from turning into a massive forest fire that takes down the whole app. Strict Time-Boxing If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. Ensures we don't accidentally leave a broken state lingering in production while we are busy looking at dashboards. Off-Peak Scheduling Schedule the breakage for 2 AM on a Tuesday, or whenever our site is basically a ghost town. If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. The "Big Red Button” Hook the chaos tool directly into our Pager-Duty or DataDog alerts. If error rates spike unexpectedly, the test auto-kills itself. It halts the damage before it gets bad enough to trigger an actual Sev-1 incident. Controlled Chaos Techniques Implementation Strategy For what Benefit Targeted Blast Radius Instead of hitting the main cluster, we target a single canary group or just one non-critical microservice. It stops a controlled burn from turning into a massive forest fire that takes down the whole app. Strict Time-Boxing If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. Ensures we don't accidentally leave a broken state lingering in production while we are busy looking at dashboards. Off-Peak Scheduling Schedule the breakage for 2 AM on a Tuesday, or whenever our site is basically a ghost town. If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. The "Big Red Button” Hook the chaos tool directly into our Pager-Duty or DataDog alerts. If error rates spike unexpectedly, the test auto-kills itself. It halts the damage before it gets bad enough to trigger an actual Sev-1 incident. Controlled Chaos Techniques Implementation Strategy For what Benefit Controlled Chaos Techniques Controlled Chaos Techniques Controlled Chaos Techniques Implementation Strategy Implementation Strategy Implementation Strategy For what Benefit For what Benefit Targeted Blast Radius Instead of hitting the main cluster, we target a single canary group or just one non-critical microservice. It stops a controlled burn from turning into a massive forest fire that takes down the whole app. Targeted Blast Radius Targeted Blast Radius Instead of hitting the main cluster, we target a single canary group or just one non-critical microservice. Instead of hitting the main cluster, we target a single canary group or just one non-critical microservice. It stops a controlled burn from turning into a massive forest fire that takes down the whole app. It stops a controlled burn from turning into a massive forest fire that takes down the whole app. Strict Time-Boxing If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. Ensures we don't accidentally leave a broken state lingering in production while we are busy looking at dashboards. Strict Time-Boxing Strict Time-Boxing If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. Ensures we don't accidentally leave a broken state lingering in production while we are busy looking at dashboards. Ensures we don't accidentally leave a broken state lingering in production while we are busy looking at dashboards. Off-Peak Scheduling Schedule the breakage for 2 AM on a Tuesday, or whenever our site is basically a ghost town. If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. Off-Peak Scheduling Off-Peak Scheduling Off-Peak Scheduling Schedule the breakage for 2 AM on a Tuesday, or whenever our site is basically a ghost town. Schedule the breakage for 2 AM on a Tuesday, or whenever our site is basically a ghost town. If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. If the system totally craters, only a tiny fraction of users will notice, keeping the business folks off our back. The "Big Red Button” Hook the chaos tool directly into our Pager-Duty or DataDog alerts. If error rates spike unexpectedly, the test auto-kills itself. It halts the damage before it gets bad enough to trigger an actual Sev-1 incident. The "Big Red Button” The "Big Red Button” The "Big Red Button” Hook the chaos tool directly into our Pager-Duty or DataDog alerts. If error rates spike unexpectedly, the test auto-kills itself. Hook the chaos tool directly into our Pager-Duty or DataDog alerts. If error rates spike unexpectedly, the test auto-kills itself. It halts the damage before it gets bad enough to trigger an actual Sev-1 incident. It halts the damage before it gets bad enough to trigger an actual Sev-1 incident. Chaos Engineering in Practice We need to start treating Chaos Engineering as an engineering habit, which can be repeatable across a structured life-cycle. Some Common Scenarios Some Common Scenarios Infrastructure: Killing EC2 instances or maxing out disks to test auto-scaling. Network: Injecting packet loss or DNS failures to validate circuit breakers and retries. Application: Exhausting connection pools or killing processes to ensure graceful degradation. Dependencies: Simulating third-party API timeouts. Infrastructure: Killing EC2 instances or maxing out disks to test auto-scaling. Infrastructure: Network: Injecting packet loss or DNS failures to validate circuit breakers and retries. Network: Application: Exhausting connection pools or killing processes to ensure graceful degradation. Application: Dependencies: Simulating third-party API timeouts. Dependencies: Automating Chaos Engineering Automating Chaos Engineering Manual tests don't scale. There is real value when we can automate this by piping these experiments to CI/CD pipelines (like GitHub Actions or Jenkins) so we validate resilience every single time we deploy. Scheduling Pattern Frequency Objective CI/CD Pipeline Gate Every single deployment. Make sure the new code doesn't ruin platform resilience. Scheduled Cadence Weekly or Monthly. Catch configuration drift and keep the on-call team on their toes. Game Days Quarterly or Ad-hoc. Get the whole team in a room, break something major on purpose, and see how fast we can fix it together. Event-Driven Chaos Real-time Trigger chaos dynamically based on specific system events or unexpected traffic surges. Scheduling Pattern Frequency Objective CI/CD Pipeline Gate Every single deployment. Make sure the new code doesn't ruin platform resilience. Scheduled Cadence Weekly or Monthly. Catch configuration drift and keep the on-call team on their toes. Game Days Quarterly or Ad-hoc. Get the whole team in a room, break something major on purpose, and see how fast we can fix it together. Event-Driven Chaos Real-time Trigger chaos dynamically based on specific system events or unexpected traffic surges. Scheduling Pattern Frequency Objective Scheduling Pattern Scheduling Pattern Scheduling Pattern Frequency Frequency Frequency Objective Objective Objective CI/CD Pipeline Gate Every single deployment. Make sure the new code doesn't ruin platform resilience. CI/CD Pipeline Gate CI/CD Pipeline Gate CI/CD Pipeline Gate Every single deployment. Every single deployment. Make sure the new code doesn't ruin platform resilience. Make sure the new code doesn't ruin platform resilience. Scheduled Cadence Weekly or Monthly. Catch configuration drift and keep the on-call team on their toes. Scheduled Cadence Scheduled Cadence Scheduled Cadence Weekly or Monthly. Weekly or Monthly. Catch configuration drift and keep the on-call team on their toes. Catch configuration drift and keep the on-call team on their toes. Game Days Quarterly or Ad-hoc. Get the whole team in a room, break something major on purpose, and see how fast we can fix it together. Game Days Game Days Game Days Quarterly or Ad-hoc. Quarterly or Ad-hoc. Get the whole team in a room, break something major on purpose, and see how fast we can fix it together. Get the whole team in a room, break something major on purpose, and see how fast we can fix it together. Event-Driven Chaos Real-time Trigger chaos dynamically based on specific system events or unexpected traffic surges. Event-Driven Chaos Event-Driven Chaos Event-Driven Chaos Real-time Real-time Trigger chaos dynamically based on specific system events or unexpected traffic surges. Trigger chaos dynamically based on specific system events or unexpected traffic surges. Tooling: AWS FIS vs. LitmusChaos AWS Fault Injection Simulator (FIS) AWS Fault Injection Simulator (FIS) If you’re already paying the AWS tax and locked into their ecosystem, just use the native Fault Injection Simulator (FIS). It’s fully managed, you don't need to install agents, and it hooks right into CloudWatch. AWS FIS Action Category Example Actions Impacted Services Instance Management stop-instances, terminate-instances EC2, EKS, ECS Resource Stress cpu-stress, memory-stress, io-stress EC2, Lambda Connectivity/Network disrupt-connectivity, inject-api-throttle VPC, RDS, Kinesis Managed Failover failover-db-cluster, zonal-shift RDS, NLB, ALB AWS FIS Action Category Example Actions Impacted Services Instance Management stop-instances, terminate-instances EC2, EKS, ECS Resource Stress cpu-stress, memory-stress, io-stress EC2, Lambda Connectivity/Network disrupt-connectivity, inject-api-throttle VPC, RDS, Kinesis Managed Failover failover-db-cluster, zonal-shift RDS, NLB, ALB AWS FIS Action Category Example Actions Impacted Services AWS FIS Action Category AWS FIS Action Category AWS FIS Action Category Example Actions Example Actions Example Actions Impacted Services Impacted Services Impacted Services Instance Management stop-instances, terminate-instances EC2, EKS, ECS Instance Management Instance Management Instance Management stop-instances, terminate-instances stop-instances, terminate-instances stop-instances terminate-instances EC2, EKS, ECS EC2, EKS, ECS Resource Stress cpu-stress, memory-stress, io-stress EC2, Lambda Resource Stress Resource Stress Resource Stress cpu-stress, memory-stress, io-stress cpu-stress, memory-stress, io-stress cpu-stress memory-stress io-stress EC2, Lambda EC2, Lambda Connectivity/Network disrupt-connectivity, inject-api-throttle VPC, RDS, Kinesis Connectivity/Network Connectivity/Network Connectivity/Network disrupt-connectivity, inject-api-throttle disrupt-connectivity, inject-api-throttle disrupt-connectivity inject-api-throttle VPC, RDS, Kinesis VPC, RDS, Kinesis Managed Failover failover-db-cluster, zonal-shift RDS, NLB, ALB Managed Failover Managed Failover Managed Failover failover-db-cluster, zonal-shift failover-db-cluster, zonal-shift failover-db-cluster zonal-shift RDS, NLB, ALB RDS, NLB, ALB You can use it to kill EC2 instances, starve Lambdas of memory, or sever VPC connections to test your database failovers. Litmus-Chaos (The Cloud-Native) Litmus-Chaos (The Cloud-Native) For Kubernetes-heavy shops, Litmus-Chaos is an incredibly powerful open-source (CNCF) alternative. It uses a cloud-native architecture based on Kubernetes Operators and manages experiments via Custom Resource Definitions (CRDs). Feature Comparison AWS FIS Litmus-Chaos Deployment Model Managed SaaS Self-managed Kubernetes Integration Deep AWS Ecosystem Native Kubernetes / Cloud-agnostic Extensibility API/CLI controlled CRD-based / Pluggable Probes Experiment Discovery FIS Scenario Library ChaosHub Public Marketplace Feature Comparison AWS FIS Litmus-Chaos Deployment Model Managed SaaS Self-managed Kubernetes Integration Deep AWS Ecosystem Native Kubernetes / Cloud-agnostic Extensibility API/CLI controlled CRD-based / Pluggable Probes Experiment Discovery FIS Scenario Library ChaosHub Public Marketplace Feature Comparison AWS FIS Litmus-Chaos Feature Comparison Feature Comparison Feature Comparison AWS FIS AWS FIS AWS FIS Litmus-Chaos Litmus-Chaos Litmus-Chaos Deployment Model Managed SaaS Self-managed Deployment Model Deployment Model Deployment Model Managed SaaS Managed SaaS Self-managed Self-managed Kubernetes Integration Deep AWS Ecosystem Native Kubernetes / Cloud-agnostic Kubernetes Integration Kubernetes Integration Kubernetes Integration Deep AWS Ecosystem Deep AWS Ecosystem Native Kubernetes / Cloud-agnostic Native Kubernetes / Cloud-agnostic Extensibility API/CLI controlled CRD-based / Pluggable Probes Extensibility Extensibility Extensibility API/CLI controlled API/CLI controlled CRD-based / Pluggable Probes CRD-based / Pluggable Probes Experiment Discovery FIS Scenario Library ChaosHub Public Marketplace Experiment Discovery Experiment Discovery Experiment Discovery FIS Scenario Library FIS Scenario Library ChaosHub Public Marketplace ChaosHub Public Marketplace A massive advantage of Litmus is its "Resilience Probes," which constantly verify our steady-state health via HTTP checks or Prometheus queries throughout the experiment life-cycle. Security Chaos Engineering: DevSecOps Matures This whole "break it on purpose" mindset is bleeding into cybersecurity, too. Security Chaos Engineering (SCE) is testing defences by simulating common, preventable mistakes. You probably will not get hacked through a nation-state, zero-day exploit. More often, breaches happen because someone leaves an S3 bucket public or misconfigures a production IAM role. SCE validates whether your controls, logging, and alerts actually fire when those things happen. Try quietly disabling a security group rule and see whether your monitoring notices. Or "accidentally" commit fake AWS keys to a GitHub repo and see how quickly your security automation revokes them. Disabling firewall rules to validate logging and alerting. Deploying an overly permissive IAM role to trace lateral movement. Simulating expired SSL certificates. Injecting dummy credentials leaks to trigger automated rotation policies. Disabling firewall rules to validate logging and alerting. Deploying an overly permissive IAM role to trace lateral movement. Simulating expired SSL certificates. Injecting dummy credentials leaks to trigger automated rotation policies. The Business Case: ROI of Resilience Building a chaos practice takes time and engineering cycles, so how do we sell it to leadership? Point to the cost of downtime. Enterprise downtime can easily burn $4,000 to $15,000 per minute. Chaos Engineering attacks that risk by reducing Mean Time to Detection (MTTD) and Mean Time to Resolution (MTTR). Teams that actively practice chaos engineering often report a 30% reduction in P1 incidents. per minute Resilience Metric Description Value Proposition MTTD Reduction Figuring out the system is broken before the roof completely caves in. Shrinks the window where customers are actually feeling the pain during an outage. MTTR Improvement Recovering from a disruption and getting things back online fast. The shorter the outage, the less money the company bleeds. Availability Score Percentage of service uptime (e.g., 99.95%) Keeps customers trusting your platform and stops the company from getting slapped with massive SLA penalties. Incident Frequency How often does the pager go off at 3 AM for a critical, hair-on-fire outage? Proves the long-term hardening of the system. Resilience Metric Description Value Proposition MTTD Reduction Figuring out the system is broken before the roof completely caves in. Shrinks the window where customers are actually feeling the pain during an outage. MTTR Improvement Recovering from a disruption and getting things back online fast. The shorter the outage, the less money the company bleeds. Availability Score Percentage of service uptime (e.g., 99.95%) Keeps customers trusting your platform and stops the company from getting slapped with massive SLA penalties. Incident Frequency How often does the pager go off at 3 AM for a critical, hair-on-fire outage? Proves the long-term hardening of the system. Resilience Metric Description Value Proposition Resilience Metric Resilience Metric Resilience Metric Description Description Description Value Proposition Value Proposition Value Proposition MTTD Reduction Figuring out the system is broken before the roof completely caves in. Shrinks the window where customers are actually feeling the pain during an outage. MTTD Reduction MTTD Reduction MTTD Reduction Figuring out the system is broken before the roof completely caves in. Figuring out the system is broken before the roof completely caves in. Shrinks the window where customers are actually feeling the pain during an outage. Shrinks the window where customers are actually feeling the pain during an outage. MTTR Improvement Recovering from a disruption and getting things back online fast. The shorter the outage, the less money the company bleeds. MTTR Improvement MTTR Improvement MTTR Improvement Recovering from a disruption and getting things back online fast. Recovering from a disruption and getting things back online fast. The shorter the outage, the less money the company bleeds. The shorter the outage, the less money the company bleeds. Availability Score Percentage of service uptime (e.g., 99.95%) Keeps customers trusting your platform and stops the company from getting slapped with massive SLA penalties. Availability Score Availability Score Availability Score Percentage of service uptime (e.g., 99.95%) Percentage of service uptime (e.g., 99.95%) Keeps customers trusting your platform and stops the company from getting slapped with massive SLA penalties. Keeps customers trusting your platform and stops the company from getting slapped with massive SLA penalties. Incident Frequency How often does the pager go off at 3 AM for a critical, hair-on-fire outage? Proves the long-term hardening of the system. Incident Frequency Incident Frequency Incident Frequency How often does the pager go off at 3 AM for a critical, hair-on-fire outage? How often does the pager go off at 3 AM for a critical, hair-on-fire outage? Proves the long-term hardening of the system. Proves the long-term hardening of the system. For SaaS and e-commerce, availability and uptime are revenue. Framing reliability as a competitive advantage makes the ROI conversation easier. How Pioneers Built It Netflix: Netflix essentially birthed the movement back in 2010 when they built Chaos Monkey. They literally wrote a script to randomly assassinate their own production servers. It was brutal, but it forced their developers to stop relying on sticky sessions and build truly stateless, resilient apps. A few years later, when an AWS regional outage completely melted down half the internet, Netflix didn't even blink. Netflix: LinkedIn: LinkedIn approaches it a bit differently with Project Waterbear. They basically treat resilience like an internal product. They use tools to intentionally make the site laggy or break specific features for a microscopic fraction of users. They lean hard on their A/B testing setup to keep the blast radius incredibly tight while they watch what breaks downstream. LinkedIn: Amazon: Before the insane traffic crushes of Prime Day or Black Friday, they don't just cross their fingers and hope for the best. They run massive, company-wide "Game Days" where they pretend an entire data centre just got hit by a meteor. It builds organisational muscle memory. When a real Sev-1 fire breaks out, nobody panics because the engineers have already drilled that exact scenario fifty times. Amazon Good Observability - A Prerequisite We cannot do Chaos Engineering without world-class serviceability. You need deep, world-class observability. If you inject a fault and can't immediately trace exactly how it cascades through your micro-services, you aren't doing science. You’re just a vandal breaking things in production. Lock down your four golden signals—latency, traffic, errors, and saturation. You have to be able to set a baseline, instantly spot the deviation, and mathematically prove the system recovered. The Bottom Line Chaos Engineering is not just a testing strategy. It is a shift in how we engineer reliability. In a world of infinite complexity, the only way to build trust in a system is to actively try to break it. By embracing failure, setting strict parameters, and automating experiments, engineering teams can stop putting out fires and start building systems that are far more resilient.