There is a particular kind of outage that nobody writes a post-mortem about honestly. The servers didn't crash. The code didn't have a bug. What happened, if you trace it carefully, is that someone in a quarterly planning meeting decided that three availability zones were extravagant, that the logging pipeline was ingesting too much, and that the auto-scaling thresholds could afford to be a little more conservative given the current burn rate. Six months later, a Thursday afternoon traffic spike finds the load balancer forwarding requests to two overloaded instances in a single zone, and the on-call engineer is paging through sparse, half-sampled traces trying to understand why latency is climbing toward timeout thresholds. The root cause isn't in the diff. It's in the spreadsheet. This is the texture of the new failure mode. Distributed, slow-moving, financially authored. Cloud-native infrastructure made reliability legible in a way that on-premise never quite did. You could see the cost of redundancy — a second replica in us-east-1b, line item, $0.096/hr. You could see the Datadog bill climbing as trace volume grew. Observability, replication, geographic distribution: suddenly, these weren't abstract engineering virtues; they were invoices. Which means they became negotiable. Which means, under budget pressure, they got negotiated away. see us-east-1b The mechanism isn't malicious. It's almost always well-intentioned. A FinOps team does what FinOps teams are supposed to do: they pull the cloud cost report, they identify the largest line items, and they build a reduction roadmap. Unused reserved capacity — gone. Verbose logging tiers — downsampled. Multi-AZ RDS — converted to single-AZ with a note that "we can flip it back before the next launch." Except launches don't wait, and the note lives in a Confluence page that nobody revisits. The reliability posture degrades silently, incrementally, in a series of individually defensible decisions. The SRE team, meanwhile, is operating on error budgets that were calibrated when the infrastructure was fuller. They're watching SLO dashboards that measure what the system does, not what the system could do under load or would do if one component failed. The budget framework — Google's contribution to production engineering epistemology, genuinely useful — is backward-looking by design. It tells you how much reliability you've consumed. It doesn't tell you that the safety margins you built it on have been quietly removed. does could do under load would do if one component failed When FinOps and SRE don't share a common vocabulary, the gap between them becomes load-bearing. FinOps optimizes for spend. SRE optimizes for error budget. Neither metric captures resilience headroom — the slack in the system that determines whether a bad Tuesday becomes a degraded hour or a P0 incident. That headroom is invisible to both dashboards until it's gone. resilience headroom There are a few specific patterns worth naming because they appear so consistently across post-mortems that they've almost become archetypes. Observability tax avoidance. High-cardinality tracing is expensive. Sampling is the standard response, and sampling is fine — until the failure you're debugging happens to fall disproportionately in the unsampled population. You're looking at 1% of traces. The anomaly is in the other 99. Teams cut observability spend because the ROI is invisible until the moment it becomes desperately obvious, which is exactly when you can no longer afford the absence. New Relic's own research on this is somewhat self-serving but directionally correct: the correlation between observability maturity and mean time to resolution is not subtle. Downsampling your way to a cheaper APM bill also downsamples your ability to understand your own system. Observability tax avoidance. The "we'll scale before we need to" fallacy. Auto-scaling is genuinely magical until it isn't. The latency between a scaling trigger firing and new capacity being ready — instance warmup, dependency initialization, connection pool establishment — can be ninety seconds to four minutes, depending on your stack. If your scaling threshold was set conservatively to save on idle compute, and your traffic spike has a rise time shorter than your provisioning latency, you're absorbing that load with whatever you have. Teams model this in theory. They rarely model it against actual traffic histograms. The threshold that looks reasonable in a planning doc looks different when you're watching the p99 climb in real time. The "we'll scale before we need to" fallacy. Single-AZ creep. Multi-AZ database configurations cost roughly double the single-AZ equivalent. The conversation always goes the same way: "Our application already does retry logic," "We've never had an AZ failure," "We can switch it back later." AWS's own availability data shows AZ-level disruptions are infrequent but not theoretical — and the failure mode when they happen is total, not graceful. The application's retry logic, written to handle transient network errors, is not written to handle the database being simply unreachable. These are different things. The retry exhausts. The circuit breaker trips. The queue backs up. Single-AZ creep. Gartner's cloud cost management surveys have tracked something interesting over the last several years: the percentage of organizations that treat cost optimization and reliability as joint concerns, with shared ownership and shared metrics, is stubbornly small. The dominant organizational pattern keeps them separated — finance-adjacent FinOps teams reporting up through a CFO chain, SRE teams reporting through engineering. The incentive structures point in different directions. Neither team is wrong, given their mandate. The mandate itself is the problem. What a careful builder would change, concretely, is the unit of accountability. Not the org chart — that's a longer fight — but the measurement framework. Introduce what you might call a reliability cost basis: for each cost reduction initiative, require an explicit assessment of what failure mode becomes more likely, what the blast radius is, and what the detection latency would be if that failure occurred post-reduction. This doesn't have to be elaborate. A structured field in the cost reduction ticket. The act of writing it forces the conversation. reliability cost basis Error budgets should get a corollary: a resilience budget, which tracks not just SLO performance but the remaining margin against defined single-fault scenarios. If removing that second AZ means you fail a simulated AZ loss, the resilience budget for that service is zero. Zero means the cost cut is not approved without an explicit exception from engineering leadership. This is gameable, like any metric. But it shifts the default. resilience budget Monday morning specifically: pull the scaling event logs for your highest-traffic services and correlate them against provisioning latency. Find the gap. That gap is your exposure window. Some teams discover it's thirty seconds. Some discover it's six minutes. Both are operating without knowing which one they are. The uncomfortable truth underneath all of this is that cloud vendors have built billing structures that make reliability feel optional in a way that physical hardware never quite did. When you owned the servers, redundancy was already paid for, or you consciously chose not to buy the second rack. The cost was lumpy, visible, and upfront. Cloud makes it fine-grained and continuous, which is mostly a gift — but it also means every line of your architecture can be individually repriced, individually value-questioned, individually removed. The granularity that enables efficiency also enables this slow, invisible erosion. Outages are economic events now, not just technical ones. The failure analysis that stops at "the instance ran out of memory" is incomplete. What it should also ask is: who approved the memory limit, when, why, and what assumption about load were they making when they did? Often, the answer is a cost reduction six quarters ago, approved by someone who no longer works there, with a rationale that made sense at the time and a risk note that said "monitor closely" and then wasn't monitored closely. Reliability isn't free. It never was. The difference is that now you can see exactly how much you're paying for it — which means you can also see exactly how much you're saving by removing it. That clarity is the trap.

Google

Cost Is Now a Reliability Problem

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Beyond the Demo: Why LLM Applications Crash in Production

Dramatic AWS Costs Optimization in 5 Steps (How We Saved 80%)

Exploring Cloud Cost Optimization with FinOps

Meet kubectl cost: A Tool For Determining Kubernetes Workloads Cost

Navigating Cloud Costs: How to Achieve Financial Visibility with Cloud Architecture

Optimizing Your Cloud Costs

Beyond the Demo: Why LLM Applications Crash in Production

Dramatic AWS Costs Optimization in 5 Steps (How We Saved 80%)

Exploring Cloud Cost Optimization with FinOps

Meet kubectl cost: A Tool For Determining Kubernetes Workloads Cost

Navigating Cloud Costs: How to Achieve Financial Visibility with Cloud Architecture

Optimizing Your Cloud Costs

Light-Mode

Classic

Newspaper

Minty

Dark-Mode

Neon Noir

Minty

HN StartUps