Why 96% of Agile Transformations Fail - Here's Why You Shouldn't Bother

Over the past 4 years, I’ve looked at millions of data points from thousands of engineering teams, surveyed thousands of software engineers and coached dozens of engineering leaders. This research has included findings that 83% of software engineers suffer from burnout, 75% of software engineers face retaliation for reporting wrongdoing at work, and 89% of business decision-makers are concerned about the on-time delivery of software projects in their organizations.

I have also investigated catastrophic software failures in the book “How to Protect Yourself from Killer Computers”, where software failures have led to aircraft entering ‘death dives’, fatal car crashes, and killer radiation overdoses in hospitals. Over the past few months, I’ve also worked with an experienced clinical psychologist and an outstanding research team including a PhD-trained psychologist and a PhD-trained behavioral scientist with experience at the University of Oxford.

This work has all been centered on the goal of understanding why some engineering teams perform better than others, and how transformations are possible.

Success rates are grim when it comes to transformations, the 2018 State Of Agile report found that 96% of Agile Transformations fail to generate the capability to adapt to changing market conditions. Even in the broader business world, when a company in England and Wales becomes insolvent, it faces only a 10% chance of survival even after the management team has been replaced with an Administrator.

Moreover, these low success rates also apply to our personal lives too. The US Government found that, in 2018, whilst 55.1% of smokers said they had attempted to quit in the past year, only 7.5% succeeded. Researchers from King’s College London studied a decade’s worth of digital health records from 278,982 people and found the chances of an obese man losing just 5% of their body weight and keeping it off for 5 years is about 1.8%.

Engineers are often the masters of change in the world; it is our role to create meaningful systems of change in the world - to provide benefits to society whilst minimizing the risks. However, even 21 years after the creation of the Agile Manifesto, studies have found that approximately 70% of software projects fail to be delivered on time.

The latest research I’ve worked on closes the circle on this work, having identified a new approach called Impact Engineering which reduces the failure rates of Agile projects by a magnitude of 6.5x whilst identifying advanced psychological strategies that turn the tide on failing personal and organizational transformation initiatives. However, to understand how this works - we must first understand how we got here as an industry.

The Agile Manifesto & Psychology

The creation of the Agile Manifesto itself is somewhat of a matter of shame for software engineers. As engineers, we have a duty to build for all of society, not merely part of it, and inform our decisions with the best information available in the science of computing and the art of programming. We have a duty to be open to engaging in public debate and be honest and clear about assumptions.

By contrast, the Agile Manifesto itself was authored by 17 white middle-aged men in Utah in a closed gathering.

An article in InfoQ has speculated on how childhood circumstances affected those who wrote the manifesto: “We found that most of the Originators grew up in homes in which the fathers were liberal professionals and the mothers were usually housewives, who had a fondness for reading, and other art- or culture-related hobbies. Most of the Originators had moved many times during their childhood and lived in several places; thus, they had to cope with changes from earlier stages of their life. Also, they were inspired at home by very similar values.”

The Psychology of Failed Transformations

Heidi Priebe, who is an author on Attachment Theory, has previously spoken about how a securely attached parent, when confronted with a child who is upset will first "picks-up, empathises with and sees the baby's emotion, but then they also help the baby figure out a constructive solution." An anxiously attached parent might solely focus on soothing the baby but not addressing the problem, whilst an avoidant parent may well skip the emotional aspects and move straight to attempting to solve the problem.

Our emotions tell us important signs before we rationally comprehend them, and they play an essential role in our decision-making abilities as multiple studies have shown those who have suffered injury to the emotional processing parts of their brains will struggle to make any decisions. Failing to account for emotional needs is a key reason for transformations to fail.

The University of Oxford’s Business School and the auditors’ EY found that even though 30% of digital transformations typically fail, “leaders who prioritise workforce emotions in their transformations are 2.6 times more likely to be successful than those who don't”.

Research conducted by David DeSteno has found that simply by someone feeling emotions of gratitude, their ability to delay gratification for a bigger reward later nearly doubles, something which happiness alone made no difference with.

Additionally, our research has also shown disturbed understandings of consent when it comes to some who evangelize transformation initiatives. Gene Kim’s “The Unicorn Project“ book project tells the story of how a software engineer becomes involved in “The Rebellion” a group trying to achieve organizational transformation by ignoring corporate rules.

Such narratives beg the question of where someone obtained this god-like status to make transformational interventions into someone else’s life without consent.

Absent from this narrative is the key concept of consent, how individuals and organizations (by their leadership teams) have the autonomy to make their own informed decisions to make mistakes (especially to the extent they don’t expose others to unmitigated risk that they haven’t consented to and the person implementing these changes is not in any kind of regulatory function).

Moreover, by depriving someone of the ability to learn from their own mistakes, you deprive them of one of the most fundamental sources of learning open to human beings. Additionally, such paternalistic attitudes disregard the fact that you may in fact be wrong about your ideas, especially in the absence of empirical evidence - and indeed the empirical evidence is shaky when it comes to Agile.

Agile on Trial

The Post Office Horizon IT Inquiry in the UK continues to investigate how IT failures, followed by cover-ups have been responsible for what

has been described as the largest miscarriage of justice in British history, linked to multiple suicides with those wrongly imprisoned including a pregnant woman.

Unknown to much of the public, the Horizon IT system was one of the earliest large-scale systems to be developed using an Agile approach, known as rapid application development (RAD). Under examination at the Inquiry, one of the Fujitsu engineering witnesses (Terence Austin) said: “one of the reasons why this got into this situation is that we were forced to do rapid application development and, by doing that, you haven’t got a functional specification.”

David McDonnell is a former Fujitsu engineering manager who became one of the earliest internal whistleblowers of the scandal and reported being “threatened with violence by an individual with a reputation for violence” when trying to address the issues in the IT system’s cash accounting module. In the end, after successfully running another project which went live without issue, he claims his employment was terminated after he refused to follow instructions "to release a piece of software that was not complete, untested and had known bugs."

McDonnell said in his own evidence to the Inquiry: “Well, a project such as that -- well, any kind of software development project, there should be a framework of how the team work. It should start with the design documents. That's the target of what you are trying to deliver; that's what you are building against.” … “I know that there had been some documents that were reverse engineered, but they were irrelevant and out of date, and they weren't even in the building when I got there. I had to ask for them.”

The Inquiry’s technical expert technical witness, Charles Cipione, summarized simply: “if you don’t have a good design, it’s not going to work properly.”

Cipione went on: “first of all, there has to be a good design process. Second, that design process has to be aligned with a very well known set of requirements from the sponsors. Only when those two things are done can you start thinking about the construction and development of the software. Otherwise, it’s very difficult to make sure that the system does what your customers want it to do.”

Empirical Evaluation

From May 3rd to May 7th, 2024, I worked with J.L. Partners to solicit responses from 600 software engineers to study how the Agile practices were working in reality. J.L. Partners is a member of the British Polling Council and abides by its rules.

Three of the four practices listed in the Agile Manifesto are “Working software over comprehensive documentation”, “Customer collaboration over contract negotiation” and “Responding to change over following a plan”. However, we found that projects that had a specification or documented requirements before development started were 50% more likely to succeed than those that didn’t.

Projects that had clear requirements before starting development were 97% more likely to succeed and projects that did not require making significant requirements changes late into the development process were 7% more likely to succeed.

Whilst reducing work-in-progress is a key tactic of those operating in Agile or Lean transformation, our findings were that it made no statistically significant difference to success rates.

However, where engineers felt psychologically safe to discuss and address problems, projects were 87% more likely to succeed and projects where the requirements were accurately based on a real-world problem were 54% more likely to succeed than ones that didn’t.

In total, those who adopted the 5 practices we identified to improve software delivery rates (which we refer to as “Impact Engineering”) were 6.5 times less likely to fail than those adopting an Agile requirements engineering approach.

Learning More

Our research has identified multiple techniques and approaches that both reduce the rate of failure in software engineering projects whilst turning the tide on failed personal and organizational transformation initiatives. Our findings are now available in a new book, “Impact Engineering: Transforming Beyond Agile Project Management”.