“Faulty configuration change”…. really Facebook? I was calling my mother when WhatsApp died on Oct 4. That in itself wasn't a bad thing as she was asking me when I will get married :) But truth be told, I waited anxiously for the root cause of the complex distributed systems issue that caused a global meltdown of 3 of the biggest software systems known to mankind, used by billions across the world. Imagine my disappointment when came out: "its root cause was a faulty configuration change on our end". Come on folks… I was really hoping for a bug in implementation of a provably-correct consensus protocol which caused a cascading failure in leader election….” this statement In some ways, this is still a step forward as at least the DevOps community uses honest words like "operator error" or "configuration mistake", which place the root cause where it resides. said it right: " " Dijkstra … begin with cleaning up our language by no longer calling a bug a bug but by calling it an error. It is much more honest because it squarely puts the blame where it belongs, viz. with the programmer who made the error. The animistic metaphor of the bug that maliciously sneaked in while the programmer was not looking is intellectually dishonest as it disguises that the error is the programmer's own creation. But the question remains: How could we still be making "operator errors" in a world where software is everything? Following Dijkstra's footsteps, let us (the community) begin by rejecting vague root causes like "faulty configuration", "operator error", etc. Someone did something which went horribly wrong. Can we get answers to simple questions: At Amazon, we used to do it through the . Wouldn't it be great if all those details are published to eventual end-users to inspire transparently? What was the change? How was it tested? How did it make it to the whole world before? What has changed to avoid a recurrence? Correction of Error (COE) process Errors are a reality so stopping at the blame won't cut it. Our community needs to think deeply about why these errors make it to production and how our tooling can be made operator proof. Graph source: https://www.kentik.com/blog/facebooks-historic-outage-explained/

Amazon

Facebook

Failure Modes: Why You Need To Know Them

Read My Stories

Too Long; Didn't Read

Get Free Unlimited IP Geo Data With IPinfo Lite

More Facebook BS: "Its root cause was a faulty configuration change on our end."

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

Failure Modes: Why You Need To Know Them

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Delving Into OpenTelemetry Collector (11/18/2023)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

105 Stories To Learn About K8s

Failure Modes: Why You Need To Know Them

The Noonification: How to Deal With Flapping or Broken Tests (11/29/2023)

The Noonification: Delving Into OpenTelemetry Collector (11/18/2023)

The Noonification: How to Implement a Merkle Tree in Solidity (11/12/2023)

105 Stories To Learn About K8s

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps