Pyramids, salt mines, or sausage factories?

Working in a modern day IT shop

Construction of the Great Pyramid at Giza. Harry Green, © Look and Learn

“Most software today is very much like an Egyptian pyramid with millions of bricks piled on top of each other, with no structural integrity, but just done by brute force and thousands of slaves.”

Alan Kay spoke these words in an interview that he gave to the ACM’s Queue magazine in 2004. Things have only become more so since then.

For this article, let’s agree that when Kay says “most software today” he is not atalking about Google, Facebook, or Netflix (especially not in 2004), and in this article I am excluding them as well. As huge as Google’s codebase is, it is a drop in the bucket. By some estimates, an equivalent amount of new code is brought into the world every week, and most of that code is below the radar.

A few hundred thousand lines added to your banking app so that you can deposit a checque by taking a photo — multiplied by all of the banks in the world.

A few thousand lines to allow a customer to process a merchandise return — multiplied by all of the suppliers in the world.

These billions of lines of code are not being written by Steve Yegge, or Chris Coyier, or Arun Gupta, or Ethan Marcotte, or Fredrik Lundh, or anyone else whose blog posts you are reading. They are being written by these developers, in this workplace…

Convergys office. Photo from Glassdoor

… and many similar workplaces around the world.

Salt mines

Kay uses the image of slaves and pyramids. The image most often in my mind is that of the salt mines. For most of human history salt was an almost priceless luxury. Salt mining — often done by slave or prison labor — was one of the most dangerous occupations in a world where life for most people was already nasty, brutish, and short.

Nobody is physically dying in these modern day salt mines, but there is a lot of misery.

In these workplaces, success is elusive, and most humans require at least occasional success to make them feel as though they are making a positive difference.

This is not trivial.

Large government and enterprise IT development projects are very rarely unmitigated successes. Serious research indicates that on the average, only 30% of projects are considered total successes by the executives who are paying for the work, and that there is another 20% that are not considered total failures (but still failures to some degree).

Now add to this the fact that successes and failures are not evenly distributed. Some (very very few) IT shops are generally successful, and account for most of the successes. For the rest; failure (whether partial or total) is the norm.

How do you think the internal discussions around these failed projects go? Do you imagine that they are pleasant for anyone? In the failing IT shops, most team meetings are about impossibles targets, missed deadlines, budget overruns, and failed deployments. I have spent most of my career in these salt mines and I don’t remember many project meetings that started with the words “high-fives all around”.

The misery is real.

In fact, true success is so elusive that on the few occasions when one of my projects was wildly, amazingly, successful, senior executives would refuse to approve a celebration because they were waiting for the other shoe to drop. Recognition would finally come many months later, with some bloodless ceremony in a quarterly divisional meeting where the presenting executive would inevitably get some important fact about the project wrong – usually leaving out one or more team members – creating a demoralizing effect far exceeding any feelings of goodwill that the tepid ceremony might have otherwise produced.

Sausage factories

I am, of course, referring to the timeless aphorism:

“Laws are like sausages. Its better not to see them being made.”

Believe me when I tell you that the same is true of enterprise software applications.

The vast majority of IT workers are not permanent employees of the companies they are coding for. Typically they work for one of several contractors who are in heated competition with each other within the account. They have the Sisyphean task of working on code that has dependancies on, and conflicts with, code that other developers — working for one of the competing contractors — are working on.

Everything breaks constantly, and everyone is pointing the finger at everyone else. It is a miracle that anything works at all, and the only way it does is by driving it all down to the lowest common denominator, and liberally applying chewing gum and baling wire (or duct tape if you prefer).

If engineers built bridges the way these companies build software, 90% of the bridges in the world would look like this:

or this:

Why?

Because these bridges are built by hand, not industrially. This is the way we build software today.

Is there any hope?

Take a look at what industrial bridge production looks like:

One day enterprise application development will look like this too; dropping in prefabricated segments that were carefully engineered to fit together perfectly. If we can do it for a 200 ton concrete slab, 90 feet in the air, we can figure out how to do it in software.

And which bridge construction crew would you rather be working on? Unless you are an atavistic lunatic with a death wish: the industrialized one.

I cannot exaggerate the levels of true despair I have witnessed in the salt mines of enterprise IT. Dozens of millions of workers suffer through every day because IT is a good living, a good way to make decent money, but they are not motivated, not happy, not proud of what the team has built.

Industrializing IT will improve the lives of millions of people in a meaningful way. And it will happen. The demand for applications far exceeds the supply of experienced and competent developers to create them. Throughout history, the solution to this problem has been industrialization, which allows unskilled or semi-skilled labor to produce high quality results. Software application will not be an historical exception.

This is going to happen.