The Cargo Cult School of Software Development

Source: https://www.flickr.com/photos/blmnevada/9607555893

This is an article about a common failure pattern at the leadership level. I’ve seen this multiple times and you’ve probably experienced it yourself. Cargo Cults are a form of corporate neurosis that needs to be diagnosed and treated as early as possible! This tendency is widespread and can emerge at any moment of crisis. And worse, all of us are susceptible to the urge to form our very own Cargo Cult…

The Scenario

A senior technology person in an organization is given a big challenge. This individual had great success in their past, using a particular model of software development. They seek to address the challenge with the approach that previously worked for them. Everyone close to the work can see that the process that “worked before” is terrible, but the proponent persists. Persists in spite of lack of support and continues to double down, even when faced with poor results and plummeting team engagement.

According to the leader, the lack of results is not due to the choice of an outmoded/inappropriate approach, not at all! The problem is that the organization is not fully recreating the approach that worked before, and needs to spend even more effort to capture all the nuances of the previous round. Now, your organization is fully in the grip of the Cargo Cult school of software development. The Cult persists far too long, resulting in enormous damage and tragic lost opportunity.

Cargo Cult Refresher

From the Wikipedia page on Cargo Cult:

“Cargo cults often develop during a combination of crises. Under conditions of social stress, such a movement may form under the leadership of a charismatic figure. This leader may have a “vision” (or “myth-dream”) of the future, often linked to an ancestral efficacy (“mana”) thought to be recoverable by a return to traditional morality.”

If the “cargo” (eg: success) does not appear, the problem is that the team is not adhering to the “traditional morality”. If we can just faithfully perform the rituals, we surely will get the same result we saw in the past!

“Past performance is not an indicator of future success”

The problem, of course, is that historical success is usually in a completely different context, and can be more the result of external historical circumstances. It is well known that people vastly underestimate how much of their success is due to random luck/timing, and overestimate their personal impact.

Early success in a career can be a curse. The most virulent outbreaks of Cargo Culting I’ve seen came from people who had great success early in their career. They enthusiastically tried to re-apply what worked in the past even though it was no longer appropriate and actually counter-productive.

What worked at Google (focusing on search) may not work so well at Yahoo…

But wait, you say: “Approach X worked great before, why not try to ride that pony to glory again?” Well, some things to consider before saddling up:

• business conditions may have changed considerably.
• users may have moved on.
• newer approaches may have come out that are more effective.
• the organizational context where approach X worked may have been very different.
• does the previous approach make sense with the culture of the company?

What is going on, why don’t people see the obvious?

Confirmation and survivorship bias are widely held cognitive biases that prevent people from properly attributing the causes of success. We humans tend to mythologize past success, and others will buy into that narrative as well. This is vividly captured by Antonio Garcia Martinez in Chaos Monkeys, his Silicon valley tell-all:

What was an improbable bonanza at the hands of the flailing half-blind becomes the inevitable coup of the assured visionary. The world crowns you a genius, and you start acting like one[…]

Leaders are expected to be “right” and to lead, yes? That’s what they were hired for. People are hired on the weight of their experience and past successes. But as this article (also about the failures at Yahoo) points out, leaders often have a big blind spot that makes it hard for them to adapt:

“Many leaders are very skilled at challenging the thoughts and opinions of others, but are woefully inept when it comes to challenging their own thinking. The reality is that it takes no effort to cling to your current thinking; however to change your mind requires you to challenge your mind. I’ve believed for quite sometime the most profound and commonly overlooked aspect of learning is recognizing the necessity of unlearning.”

Bluntly, people are mentally lazy. It’s vastly easier to maintain our mental model than question it.

Source: https://pixabay.com/p-255423/

Another factor is that sometimes people, in their climb to senior roles, have been involved primarily in organizational and bureaucratic areas. After a period, they will have fallen out of touch with advances in the practice of software product development.

When I started writing this post, I wanted to savagely criticize this behavior. I’ve seen this pattern cause enormous damage to organizations, and thinking about all the wasted potential still fills me with righteous, indignant rage.

But on reflection, I came to sympathize with this mindset. And I realized this behavior is something we are all prone to. We’ve all had times when the shit hits the fan, and we revert, in panic mode, to “what we know works”. In retrospect, it’s easy to see that one should have taken time to think about the challenge before heedlessly diving in. It’s one thing, however, to exhibit this behavior as an individual. It’s epically more destructive to subject the team/organization to an individual’s cognitive blind spots.

And there’s a bigger problem underlying these scenarios: the higher-level organization failure that allows this type of behavior. The company that continues to empower failing (flailing?) leaders is even more at fault than the all-too-human leaders who are falling back on what they know.

The organization shortcomings that allow Cargo Culting to develop are largely around a lack of accountability and honest 2-way communications. One of my all-time favorite essays describes the anti-pattern of The Thermocline of Truth. In this model, the true situation of a project (things not going well) is well known, but a number of factors prevent the reality from rising to senior management. Eventually, the truth bubbles up to the top as a very unpleasant surprise. But by this time, it’s way too late and damage has already been done.

Contributing factors to a Thermocline of Truth:

• a lack of meaningful metrics
• unchecked optimism by developers/product
• managers prefer to focus on the positive and understate the negative
• upper management tends to reward good news and punish bad news

A healthy organization will have ongoing validation rolled into all significant decisions. When I was working in my first real agile product development role, I came in with an agency background, and I was honestly in a bit over my head. My background was seriously lacking in terms of thinking about testability, configuration management, deployment considerations, etc. And I had a tendency to exuberantly cut corners and rush to a solution, which was completely kosher and appropriate in my previous jobs. In short, my default approach was wildly inappropriate in my new context.

But, the technology team there had a culture of vigorous feedback and I was firmly and consistently nudged in the right direction. My Cargo Cult of “that’s how we did it back in the agency” never had a chance to take root.

Recognizing Cargo Cult Development

This is dead easy! You will see all of these signs around a Cargo Cult:

• The champion of the Cargo Cult is pushing an approach they had great success with in the past.
• The people closer to the work (developers, product) are constantly asking “Why are we going down this path?” (eg: a Thermocline of Truth has formed)
• The team is demotivated and unengaged
• The path is maintained in spite of ongoing poor results
• There is no feedback loop for course correction
• All significant decisions made behind closed doors

Tech Leaders: How to Avoid Starting a Cargo Cult

If you are in a crisis mode, you are likely to fall back into your “go-to” patterns, and that may not be the best path. Consider some of the following points before reverting to “what you know works”:

Your assumptions are probably wrong, so it’s better to not make big bets. Apply the validated learning approach.

Always use a quantitative indicator to track success.

Keep an open mind, allow room for unlearning.

Consider opening up the circle of decision makers and take advantage of the collective IQ of your team.

Maintain a growth mindset. Invest the time to keep up with advances in software and product development.

Listen whole-heartedly to critical feedback, especially if you keep hearing repeated messages.

Keep an eye on team engagement. If your people have lost their sparkle, dig into the reason.

Cultivate humility, per Richard Feynman, from the Wikipedia page on Cargo Cult Science:

“Feynman cautioned that to avoid becoming cargo cult scientists, researchers must avoid fooling themselves, be willing to question and doubt their own theories and their own results, and investigate possible flaws in a theory or an experiment.”

Surviving a Cargo Cult

If you find yourself living under a Cargo Cult regime (and you probably will at some point), I’m sorry to say that your options are not appealing. The organizational flaws that allowed the Cargo Cult to flourish make it very hard for someone lower in the power structure to change conditions. Here are the dismal paths to consider:

1. The Cassandra Option

You can try and become the loud advocate of speaking truth to power, but this is likely to imperil your career in the organization. It’s nice to be on the right side of history, but if the organization doesn’t want to hear/see the truth, you will be seen as the problem.

2. Wait for the fever to burn out

If you believe in the organization, you can try to ride it out, and find space to do the best you can within the system. The company will come around eventually, and it may be worth waiting for that. You have to also consider how much a long period of following a discredited system will impact your career.

It’s also very hard to remain untouched by your environment. Working in a bad system will inevitably deteriorate your skills, and you are learning bad practices that you’ll have to unlearn later on.

You might be lucky enough to maintain or build a bubble of excellence and rationality, and continue growing your career. In a bigger organization you might be able to transfer to a point of safety outside the eventual blast radius. But I’ve found that Cargo Cults tend to gradually encompass everything and any refuges of sanity are eventually bulldozed in the name of “alignment”.

3. Get out

The higher placed the problematic leader is, the more mandate they have to do things their way. And the longer it will take for the company to lose patience with the ongoing hot mess. Inevitably also, the blame for the failed approach will be distributed widely and you will probably get a good portion of it just by being around.

Recommended action: hunker down or leave — don’t expect change while the beloved leader is empowered.

Closing Thoughts

There are some models of software product development that work remarkably consistently, particularly what I think of as “the Amazon model”. That is: a fully cross-functional product team (< 10 people) that is responsible for business results, empowered, and owns everything from ideation to deployment. That said, this structure would not be appropriate in every situation, for example in an early stage startup.

If you got this far and enjoyed what you read, please take a fraction of a second and click that little heart down there on the left!