“Resetting Norms” in Software Development

“The gradual process through which unacceptable practice or standards become acceptable.” Does that sound familiar? It happens on most teams. Your code base is eroding and you are accumulating technical debt. Your quality is gradually slipping. Your processes are becoming more bureaucratic.

The term “normalization of deviance” comes from an examination of the Challenger disaster at NASA, and has also been applied in medical fields. In those cases, the deviance ultimately leads to a disaster (of the “lives lost” kind). Most of us aren’t in the business of building rockets or saving lives, but we still all normalize bad things.

I first read about normalization of deviance in the context of software development in this article by Dan Luu. Of course, there are other theories around norm-setting and behavioral psychology, like the broken window theory (usually applied to crimes and urban environments), but it usually boils down to the same thing: the presence of Bad Things creates a norm that allows the creation of other Bad Things, often with a slippery slope where the Bad Things get worse. As social, adaptive creatures, we quickly normalize the behavior of those around us — good or bad. And once we’ve normalized things, and they’ve become part of our culture and our processes, they are really hard to change.

Sometimes, these normalizations are mindless and not intentional (for instance, imitating someone else’s bad code pattern without understanding why it’s bad). In other cases, people are doing what they think they can get away with because they see others getting away with it (for example, I know I should write unit tests, but my colleague just pushed some untested code). You can actually get deep into game theory and motivational theory if you really explore what’s happening. For instance, if my goal is to maximize my productivity relative to my colleagues, then I’m incentivized to spend as little time documenting my code, while hoping my colleagues do the opposite — but we might end up with a Tragedy of the Commons.

Dan Luu has some recommendations to combat this normalization — “paying attention to weak signals” and making sure you set up the right incentives. Hidden in one of his foot-notes is another recommendation, which I’d like to expand on: “asking about why things are the way they are”.

Asking “why” is an “abstraction barrier crossing”. Like all abstraction barrier crossings, it requires deeper thinking and can be mentally exhausting. The barrier you’re crossing is if “I accept things the way they are, and take them as immutable, it’s easier to reason about the work I need to do”. That’s a great simplification when you’re executing on a plan, but crossing it is necessary if you want to debug culture or process.

Crossing it is a lot easier if you “reset your norm”. It’s a lot easier to convince yourself and your team that things should change if you can point to other concrete examples. Here are a few tricks:

• Normalize externally. There’s a few ways you can do this. You can go the anecdotal route, and talk to or read about other companies or teams and how they compare to your team. For instance, once you learn that for some teams at Google, any incident that requires manual intervention results in a post-mortem, it might help you realize that the fact that your team has normalized several incidents each day. Maybe learning that Quora still does continuous deployment and post-commit code reviews even with >100 engineers can make you wonder why your deployment process takes hours or even days. Or if it’s something measurable, you can benchmark against other companies or industry standards. Of course, just because someone else does it or can do it, it doesn’t mean you should (you should always avoid Cargo Culting). This exercise just helps you reset or expand your view of what’s normal and possible.
• Normalize historically. If the main reason deviance creeps in is because it happens gradually, you need to look at a longer timespan to avoid recency bias. One quick way is to ask people who were on the team a while back how things were and compare that to now, or, if it’s something you can measure, you can just look at how that measurement has changed over time (e.g. we used to have 2 high-priority bugs per week, now we have 20). Again, what was possible historically may not be possible or may not make sense now, but it’s still good to do the exercise.
• Do a “from-scratch” exercise. If nothing (code, process, team) existed and you were building your system or team from scratch, what would it ideally look like? How does that compare to now? For this exercise, it often helps to talk to people with a fresh mind, either because they are not part of the team, or because they very recently became part of the team and haven’t normalized the state yet. In fact, I usually ask anyone joining a team I’m on to keep a written list of things they note in their first couple months, and then revisit it a few months later to see what they’ve normalized.
• Document your target norms. Having your goals written and easily-accessible makes them easier to restore and easier to maintain.

Finally, in addition to creating norm-resetting channels, it also helps to have channels for people to fix any instances of deviance. You might be able to rely on managers and leaders, but a lot of times, the best fixes are driven from the trenches, and that requires a culture that enables people to become “champions” of a cause.

I came across the term “champions” in the book In Search of Excellence. The idea is that you need to create a culture and processes that enable “those individuals who believe so strongly in their ideas that they take it on themselves to damn the bureaucracy and maneuver their projects through the system.” For instance, Google created 20% time and “grouplets” as a channel for people to champion causes. At Quora, we use a concept of Directly Responsible Individuals (aka “DRI”s, which I think was borrowed from Apple). For instance, we might appoint an engineer as company-wide “Code Quality DRI” or “Onboarding DRI”. Additionally, as managers, we try to coach ourselves to identify when someone is a potential champion, and let them contribute to or own solving the problem (instead of saying something like “oh, thanks for pointing that out, I’ll see what I can do about it”, we’d say something like “Have you thought through possible solutions? Would you like to be involved in solving this?).

Have you had success “re-normalizing” things on your team? I’d love to hear more!

Thanks to Ahmed Medhat and Ehab El-Badry for discussions and suggestions to this piece.