Agile: the greatest struggle that never existed

Not your granddad’s waterfall

I have been working with an Agile development shop of late, and there is something about it that I definitely do not like. I do not enjoy being “educated” by well-meaning but nonetheless smug Scrum Masters and Product Owners who assume that anyone in corporate IT has not the slightest clue. These are not ad hoc lectures, they are formal presentations; Powerpoint decks approved by the development shop’s executives. The implication is that the company assumes we have never even heard of Agile.

The reality is that many people in corporate IT are as familiar with Scrum (because that is the only kind of Agility anyone seems aware of these days) as they are. And some, like myself, have significantly more experience, having worked with other Agile practices such as XP, CRC cards, Crystal, et al. I started Agile practices (XP in 1996) before they were officially “Agile”, and for that matter before some of these Scrum Masters were born.

This attitude of smugness is nothing new, I have been more or less continually confronted by it since late 2004 when Ken Schwaber published his paper on the “SCRUM Development Process” which triggered a new round of holy wars in the field of software development.

Or did it…?

This holy war was very one one-sided. I haven’t seen any significant opposition to the basic ideas of Scrum. What I have seen is two things:

1. Annoyance with the holy fervor and maddeningly preachy attitude of the Scrum faithful, and
2. The requirement that anyone coding this way still conform to PPM and Stage-Gate governance.

It is this last point that seems to be radically misunderstood by the Scrum contingent. They seem to believe that CFOs, Business Sponsors, and Project Managers actually give a rat’s ass about how they write code.

Believe me: nothing could be farther from the truth. I can easily name a dozen common practices that would immediately cease if corporate IT management knew the least thing about (or truly cared about) how the sausage gets made.

But what is Agile, really?

I consider the concerns of Agile to be orthogonal to the concerns of governance strategies such as PPM and Stage-Gate, not in direct conflict to it.

I have been careful to use the word “practices” rather than “methodology”. As several of the original signatories of the Manifesto for Agile Software Development (Dave Thomas, or Bob Martin or Martin Fowler or Ron Jeffries) are at great pains to point out: the original impetus for Agile practices were concerns about the craftsmanship or practice of software development, not the process of software development. The manifesto itself says: “Individuals and interactions over processes and tools”.

Yet most people “doing Agile” today have this idea that they are the sworn enemies of the oh-so-evil waterfall process. It stands to reason that they also believe that Agile is a process.

And what does Waterfall really mean?

What most Agile developers think of as “waterfall” is actually a combination of two very widely adopted models of governance, the first being Project Portfolio Management and the second being the Stage-Gate Model.

The first and most important thing to understand about these two models is their reason for existing. They exist to protect corporate executives. They do this by demonstrating good-faith best-efforts to comply with two important Securities and Exchange Commission (SEC) regulatory requirements.

The first of these is the American Institute of CPA’s Statement of Position (SOP) 98-1, which was passed as a result of CFOs seeking SEC approval to account for software development costs as capital expense (CapEx) because that would allow them to report higher earnings and higher company valuations.

The second is the Sarbanes–Oxley Act of 2002 (SOX), which had been passed in reaction to a number of major corporate and accounting scandals, notably Enron and WorldCom (who manipulated earnings, underreporting line costs by capitalizing rather than expensing).

To make a long story short, corporations are allowed to count software development as CapEx, but the officers of the company will be criminally liable if audits discover that this has been done improperly.

PPM and Stage-Gate were promoted by Gartner and the Project Management Institute respectively, as governance that would protect corporate officers from criminal liability as regards the capitalization of software development costs.

So what?

So… the concerns of Agile and the concerns of “waterfall” are quite separate, there is no reason why they would not co-exist peacefully. All that is required for a Scrum team to comply with almost any modern IP project governance (PPM and Stage-Gate) are six artifacts:

1. a Project Charter which is basically just a scope document and is roughly equal to the Product Backlog.
2. a WBS (work breakdown structure) which can usually be satisfied with the results of Epic-level Planning Poker combined with a rough Sprint Planning
3. weekly Status/Progress Report is the weekly burndown report or any other form of progress report
4. weekly Open Risks which may not be a standard Scrum artifact but there certainly is nothing about this that is problematic in Scrum
5. weekly Open Issues which are a direct result of Sprint retros and therefore standard in Scrum
6. Lessons Learned which are also a direct result of Sprint retros and therefore standard in Scrum

There is nothing here that is particularly antithetical to Scrum.

Traditional Stage-Gate separates “detailed design” from “development” but it really is no big deal from the perspective of the main concern: both are admissible as CapEx, so it is materially* inconsequential whether or not they are performed sequentially, concurrently or iteratively.

* I use the the word “material” in its accounting sense

Conclusion

I would urge Agilites to stop positioning yourselves as the “sworn enemy” of waterfall, you will not win. Corporate executives do not want to go to jail, and there is nothing in Agile or Scrum to protect them from that. PPM and Stage-Gate are not going away, learn to live with them.

Consider dialling back on the pre-project Scrum evangelization and focus on making things as easy and as transparent as possible for your enterprise customers, particularly the Project Management Office (PMO).

The most evangelical thing you could possibly do is simply deliver a functioning software product on-time and under-budget without creating any waves in the PMO. Just do that, and your customer will gladly be all Agile, all the time.

Actions speak louder than words.