We often think of Software Development as a pursuit grounded in logic and clarity. There are stereotypes of software developers as devout followers of logic, more interested in binary digits than their fellow humans. From this perspective it can be shocking to discover that software projects have high failure rates and that some projects fail to the tune of billions. How is this level of chaos possible in a logical discipline with scientists at the helm?
If we can better understand how projects are so subject to failure then we can learn to approach projects more safely in future. The most dangerous risks are the ones we don’t see coming - especially if our way of thinking blinds us to them.
Stereotypes about Software Development
Our image of Software Development as a logical pursuit is exemplified in the precise and strict syntactical rules of programming languages and applications composed from them. This pervades popular understanding as well as technical thinking. The rules of some applications can be clear to the user of the application when using it - simple games like space invaders typically have no page explaining the rules as we simply see them. Other times it may be clear an application has rules but what they are can be obscure, resulting in the familiar ‘computer says no’ phenomenon. Either way the strictness of the rules tends to be felt and this influences our assumptions about software development.
Famous examples of computing problems are also a big part of our picture of software and these tend to be very strictly defined. Travelling salespeople in textbooks are subject to more constraints than in real life (they rarely get lifts or share jobs with colleagues or bunk off for a drink). We tend to heavily associate software engineering with examples about breaking secret codes, not so much with software that helps assess insurance claims.
But software developers are not all academics and do not all work on well-defined problems. Professional software development is about codifying rules to produce a system that meets a set of business objectives. It aims to produce artifacts of logic that will serve purposes in a messy world.
So, despite the pervasive picture of software development as strictly logical, the messiness of the world does have an important hand in the process of making software - software is made to play a role in the world. Projects can go wrong by making incorrect assumptions about the world or being confused about what they are trying to do.
Even so, this is only a partial explanation. How does it come about that software projects can be so risky and fail so often? How do we go about writing software?
Pictures of Software Development
Software projects can vary immensely and the lifecycle of how projects are implemented can vary a great deal. It’s quite tricky to form and hold a clear picture in one’s head of the process of making software. It varies so much that to an extent one simply has to learn from experience. But having a clearer picture of software development will help us better appreciate the sources of risk in software development.
Given how abstract software development is it can be tempting to fall back on analogies to more concrete disciplines. Especially tempting can be an analogy to construction or physical engineering. So tempting is this analogy that it has historically influenced a great deal of software project management methodology (especially on the waterfall model). But we should be cautious about this:
I'm very suspicious of using metaphors of other professions to reason about software development. In particular, I believe the engineering metaphor has done our profession damage - in that it has encouraged the notion of separating design from construction. Martin Fowler
As Fowler points out, it is dangerous to think we know in advance what the problems are.
It is also dangerous to think we know in advance what the tools are. It is tempting to think that programmers write code and therefore the primary tools are programming languages. But contemporary commercial development relies a lot on existing tools and libraries which have to be chosen and applied to the task at hand. Programmers don’t just write code to solve problems - programmers do a lot of writing code to stitch together the partial solutions of others. Depending on the choice it may be difficult to make a tool work for a particular purpose (requiring a lot of thought and stitching code) or routine or it may be practically impossible.
Construction also involves problems of tool selection. For example one has to choose types of brick or material for load-bearing and durability. The difference with software is the tools themselves are conceptual so the boundaries of what they do are harder to see clearly, at least without devoting very considerable time. It is not unusual to discover down the line that a chosen tool doesn’t support the particular data format or communications protocol that we wanted it to.
Unchallenged early assumptions about tools can certainly lead to overruns and cost increases, which can contribute to failures. More damaging though tend to be unchallenged assumptions in the space of the business objectives. Here again the construction metaphor doesn’t reflect the risk. There is certainly scope for mistaken assumptions to slip through about whether a building design provides enough space for the number of people who will use it and whether there are enough elevators. But visual plans help to bring out assumptions and the more concrete the realisation becomes the easier the assumptions are to find. Since software is abstract by nature it is difficult to make it concrete enough to flush out assumptions without entirely building the solution (by which point all the cost has been incurred).
Refining problem specifications to flush out assumptions can be more art than science. Refining specifications and choosing tools to address them are both central to what software developers do. Risk is inherent to both. Let’s try to articulate a metaphor that expresses this better than the construction metaphor.
A Better Metaphor
Software development is both logical and creative (and therein lies its chaos). There is a particular class of problems which are both creative and logical, a class which might be thought to fall under the heading of ‘riddles’ (though I do not think everything called a ‘riddle’ is necessarily a good fit). We can explore how creative problem-solving of this kind can serve as a metaphor for software development by considering a particular story of puzzle-solving from the old Norse saga of Ragnar Lodbrok.
As the story goes, Ragnar’s men report to him that they have seen avery beautiful woman, Kraka. Ragnar’s interest is aroused and he sends for her, but he decides to test her wits. He commands her to arrive neither dressed nor undressed, neither hungry nor full, and neither alone nor in company. Kraka is up to the challenge and arrives draped in a net and her long hair, biting an onion, and with only her dog as a companion. Ragnar is impressed and marries her.
Ragnar’s request is not unlike the specification of a software project in at least one key respect. Ragnar does not entirely know what will satisfy his request until he sees it. He knows certain constraints that will need to be met but he will have no idea that the solution Kraka finds is a possible solution until he sees it.
To see the connection more clearly, let us imagine that Ragnar does not immediately marry Kraka. Instead he next specifies that Kraka should first prove her worth in an even more heroic exercise. He asks her to produce a system which will process the documentation approvals and rejections for which he currently employs a whole team. At the point of specification, Ragnar has little conception of how that task might be achieved without the need for a team.Much as with his previous riddle, Ragnar does not know it could be possible to process the documentations approval and rejections without a team. He needs to be shown the solution before he can know whether it does what he currently calls ‘documentation approval and rejection.’
It is important that in both cases Kraka has to take Ragnar into a kind of unknown territory. Ragnar is not sure of what things he would call ‘neither dressed nor undressed.’ Similarly, Ragnar is not sure what he will call ‘documentation approval and rejection’ without a team performing the approvals/rejections and Kraka has to understand these expressions sufficiently to come up with something that Ragnar will call a solution.
The basic uncertainty that goes along with software projects, then, is that we do not fully understand what the solution to our problems will be until we have a completed solution. If all goes well then one sees the solution take clearer and clearer shape as one goes through the project.This is the key difference between software projects and manufacturing projects –with a manufacturing project one can form a relatively clear picture very early on of the product to be produced and then it can be precisely drawn up and physically constructed. In a manufacturing project one has the possibility of forming a clear picture early on because the product to be produced is a physical thing. A software product is abstract and the solutions to software problems are conceptual. (Software programs may be physically instantiated but this does not take away from their conceptual nature.)
If one is drawing up requirements for a physical product or structure, those requirements are by and large much easier to understand from the beginning than is the case with a software project. For example, if one wants to build a bridge then it is fairly clear from the beginning what it means to say that it needs to take a load of X many cars each weighing so much. Software requirements, by virtue of being abstract, are more likely to be hard to understand and can sometimes look like they are perfectly clear but turn out to be unclear in the light of later developments.
Let us go back to Kraka and Ragnar and imagine that Kraka arrives accompanied by a rabbit instead of a dog. Ragnar might say that he will not accept this as a solution – he thinks that arriving with a rabbit does not count as being accompanied at all. If this seems implausible or unfair then perhaps imagine that Kraka arrives with a goldfish instead. It is plausible that we can find circumstances where Kraka and Ragnar will take different views of what counts as being ‘accompanied.’ But it is Ragnar’s view that counts since he set the problem.
A parallel of the dispute between Kraka and Ragnar takes place on software projects - developers can sometimes take a different view of what fits the problem to what end-users will find acceptable. Kraka could feasibly have sought to forestall such an issue by probing the problem further from the beginning – she might have done an equivalent of requirements analysis. We can imagine Kraka putting a business analyst hat on and asking questions like ‘Do you mean accompanied by a human being? What about a pet?’ But even after much requirements analysis these types of ambiguities can remain. Even if it does become clear that Ragnar would accept a pet, it may not be made clear (not even to him until he sees it) that he would not accept a goldfish.
Uses of the Metaphor
I personally find the logical problem-solving metaphor a useful fallback when I'm tempted to look for a definite answer to whether a particular design will 'work' or a chosen tool will fit a particular need. Sometimes we can't be sure of a solution in advance of providing it. We might very much want that upfront certainty but we just can't achieve it.
I also find the metaphor useful when there’s a lot going on in a project and I need to better see the project dynamics - how the project is operating and where it is trying to get to. Thinking of a project through the open-ended comparison of logical problem solving can help me see the importance of aspects that I might otherwise overlook. It can give me a rationale to better understand the dynamics of a project rather than seeing a set of burn-down charts. If we see software projects as exercises in collective problem solving then we refocus our thinking away from the graphics of plans and instead towards the people and motivations driving the project.
This article is based on my 'Why Making Software is so Difficult', ACM SIGSOFT Software Engineering Notes 39