Preventing serious faults is within reach TL;DR: Use mature tools to make mature software. The Problem On July, 9th 2022, bricked several servers. yet another hardware failure Looking at failure's root cause we can learn a lesson. On we can find what happened: this thread I once had a small fleet of SSDs fail because they had some uptime counters that overflowed after 4.5 years, and that somehow persistently wrecked some internal data structures. It turned them into little, unrecoverable bricks.
It was not awesome seeing a bunch of servers go dark in just about the order we had originally powered them on. Not a fun day at all. https://www.techradar.com/news/new-bug-destroys-hpe-ssds-after-40000-hours The Cause The fault fixed by the concerns an Assert function that had a bad check to validate the value of a circular buffer’s index value. Instead of checking the maximum value as N, it checked for N-1. The fix corrects the assert check to use the maximum value as N. Dell EMC firmware The Prevention We are serious software engineers and we have mature tools. How can we prevent defects like this one (They are not ) BUGS . Stop Calling them 'Bugs' 1 - TDD With , we can only write code after a failing test. TDD In this way, we need to think of the N scenario and explicitly check the case. Otherwise, we cannot write code. TDD is incredibly good for . embedded systems 2 - Zombies is a great testing tool and also an amazing TDD companion. Zombies The 'B' for Boundaries at tells us to explicitly check for border cases. zomBies In this case N - 1, N, and N +1. How I Survived the Zombie Apocalypse 3 - Mutation Testing Whenever we use arithmetic or , we might check what would happen if we make a mistake (like the one in this article) and change a < for a <=. IF conditions Mutation testing is a very powerful tool to check boundary scenarios. 4 - Model Circular Buffers Embedded and hardware systems are often tuned for optimal performance. They skip some checks and are programmed with low-level languages. Most of them avoid the real world and use short integers as indices. MAPPING According to our , an is not a (or ) and a is not an . MAPPER integer shortint longint shortint integer 5 - Fail Fast Mission Critical software sometimes has recovery or fault-tolerant routines. Following principle, we can anticipate disaster and let another piece of code take over instead of bricking the disks. Fail Fast Conclusions. Systems don't fail. We fail as software engineers and make the same mistakes over and over again. Also Published here

Amazon

BUNCH

Dell

Twitter

How to Find the Stinky Parts of Your Code [Part XXI]

GPT-3 Training Programmers for the Present (and the Future)

Buy My Book

Share your ideas with me!

Read My Stories

Buy Me a Coffee

Too Long; Didn't Read

Make resilience your competitive advantage

5 Ways to Prevent Server Hardware Failure

5 Ways to Prevent Server Hardware Failure

About Author

Comments

TOPICS

THIS ARTICLE WAS FEATURED IN

Related Stories

Untitled Story

10 Predictions About the Future of Finance

106 Stories To Learn About Clean Code

100 Pieces of Programming Advice from the Book Clean Code by Robert Martin

110 Stories To Learn About Refactoring

3 Things Coding And Prose Writing Have In Common According To Cory House

10 Predictions About the Future of Finance

106 Stories To Learn About Clean Code

100 Pieces of Programming Advice from the Book Clean Code by Robert Martin

110 Stories To Learn About Refactoring

3 Things Coding And Prose Writing Have In Common According To Cory House

Light-Mode

Classic

Newspaper

Dark-Mode

Neon Noir

Minty

HN StartUps