How to Get Better at Estimating Software Development Time

Being able to accurately estimate the time required to develop a given product or feature is a crucial skill for every developer and one that must be honed over time through deliberation and effort. As an industry, we are constantly plagued with software overruns. One study estimates that as many as 66% of all enterprise software projects have cost and effort overruns.

Estimates permeate through all stages of the development process. Projects are typically drafted with time bounded constraints. The project may then be broken up into specific milestones, each with a designated deliverable date. Further still, the developers would likely use some intuition to break up the tasks into more manageable chunks that can be completed in shorter intervals of time, i.e. within a day, half a day, a week etc. A setback in at any one of these stages can ripple upward causing the entire project to gradually go off schedule.

Very often, our estimates of certain tasks are guided by personal heuristics and past experiences . If you’ve worked with a piece of technology in the past, you are more likely to be aware of it’s intricacies and potential complications. This makes you better equipped to effectively estimate the time needed for development.

But on the flipside, as software people, we are also constantly searching for new challenges and working with domains that we may have little to no experience in. In these circumstances, using past experiences can not only be misleading, but also counter-productive.

Similarly, on a personal level, your productivity may very well vary from day to day. Certain days are more productive than others, allowing you to get more work done. Recognizing this allows you to better plan your work day/week and get a better handle on how much you can realistically accomplish.

Nonetheless, like all decision-making, our intuition is also prone to some level of bias. While this cannot be completely avoided, being aware of the potential blind spots that exist and knowing how they can be avoided definitely helps one improve the estimation process. Here are some of them:

1. Building Software is a Process of Discovery

One thing that sets software development apart from other engineering related disciplines is how much room there is for requirements to evolve as the project progresses. This is, in large part, thanks to the intangible nature of a software product. Requirements aren’t always set in stone at the outset and even if they are, there’s usually some leeway to modify them later on. Making changes to the code at a later stage is often perceived to be easier than it really is.

In some sense, this is a variation on the age-old adage, “Customers don’t know what they really want until they see it”. This is particularly true for user facing products, where users often discover what they really want from a product after they’ve started using it, or playing with it — causing requirements to change.

This is one of the reasons for the broad proliferation of agile inspired development processes used by several organizations. The goal is to incorporate healthy doses of customer feedback throughout the development process to get a better sense of where the product is headed and write code in a way that minimizes waste later on. This obviously has the added benefit that development estimates can be revised frequently, in step with changes in the product requirement.

For projects that are not necessarily user-facing, there can still be some value in frequently communicating the requirements with the team and staying on top of any upcoming changes — planned or otherwise.

2. Parkinson’s Law : Work Expands to Take The Time Allocated to it

Most of the discussion so far has revolved around identifying and minimizing potential underestimates in the software development process. While this is obviously extremely important, it is also equally important to understand the problem of overestimating the development time for a particular product or feature.

In an effort to avoid the scenario where something was promised but not developed, estimates are often generously padded to allow for any unforeseen delays encountered along the way. Better to under-promise and over-deliver than to over-promise and under-deliver, correct?

The problem with this approach is that a lot of times, if you allocate, say an entire day for a particular task, you are likely to end up consuming the entire day even though you could have completed the task in a less amount of time. It is purely psychological, but also highly consequential. Even though the results may not be very drastic, there is some merit to the fact the simply giving yourself more slack is not necessarily the best way to mitigate poor estimates.

Fortunately, a lot of teams already recognize this problem. This is, in part, what has led to the adoption of Agile-based “sprints” where work is divided on a recurring weekly/bi-weekly cycle. Every one to two weeks, the team regroups and comes up with a new roadmap for the next cycle. Keeping cycles short has a tremendous impact on the quality of estimates since there is more immediate feedback.

In general, tasks must be kept as granular as possible to get a better handle on how much time is needed to complete them — both as a team and as an individual. Larger tasks and projects with deadlines far into the future are notoriously difficult to assess.

3. Context Switching

A scenario that comes up quite frequently as a developer is the need to balance multiple independent projects simultaneously. This is usually unavoidable and is also usually quite desirable. Project schedules are unpredictable and one can often get “blocked” at a certain stage while waiting on other members of the team. In such situations it is in everybody’s best interest that there is a pile of tasks or an alternate project that needs to be worked on to best utilize the developer’s time.

But frequent switching between tasks also comes with a very real cognitive cost. This is because programming can be very intellectually demanding and requires the programmer to be in the right headspace a.k.a “the zone”. The zone is an elusive state of mind where productivity is heightened and large amounts of work are pumped out through sheer concentration. Only caveat is that it usually takes some not-so-insignificant amount of time to achieve that level of flow and, once broken, it’s very hard to regain.

Frequent transitions between tasks throughout the day, along with other interruptions, thus come with the consequence of reduced overall productivity. It is generally much better to set aside pre-determined blocks of time for each task, with each block being long enough to actually be productive within it.

And more importantly, at every point, try to ensure that you have a good mix of work that is of higher priority and is likely to be blocked less often as work that is vaguely defined and that you can work on in your downtime.

4. Estimates are not transferable between team members

An important thing to realize is that programmers vary widely when it comes to talent and experience — both in a general sense, and with respect to specific languages and technologies. Someone with more experience in a related area is likely to have a widely different perspective on how difficult a task is than someone from a completely different background. It is, therefore, important that estimates are developed keeping every developer’s individual skill levels in mind.

Better estimates can be achieved by building off off everyone’s experience. Involving more people in the estimation process gives developers a better idea of the kind of challenges to expect. Gamified approaches such as Planning Poker are also quite popular and to do a good job at making everyone’s opinions heard.

5. Communication Overhead

Lastly, when it comes to effective estimation of projects, one must also account for all the time that is consumed in the form of communication overhead. This holds particularly true for teams of a larger size.

One of the key arguments presented in the classic Mythical Man Month is that the addition of more developers on a team, beyond a certain point, has the inadvertent effect of slowing it down. This is primarily due to the considerable amount of time that has to be spent “ramping up” the new person on the team and getting them familiar with the codebase.

Not only is the new person unable to contribute significantly for a while, but this also takes away precious time from the people involved in getting them educated. This makes it harder to assess the overall progress of the project and make estimates since resources are being diverted away so frequently.

On a similar note, adding more people to a team also exponentially increases the number of communication links. For projects that are highly collaborative and have a lot of people working on overlapping components, it is absolutely critical that everybody be on the same page. As such, keeping everyone informed and updated likely introduces some form of overhead, primarily through meetings. As the number of people increases, so does the communication overhead, and with it the estimation uncertainty.

Obviously, one solution is to keep team sizes small and reasonable, generally 4–5 people. For teams that are larger, work should be distributed between sub-teams that have minimal overlap.

Additionally, while not a complete replacement, effective asynchronous communication tools such as Slack/Chime/MSTeams go a long way in minimizing the time spent in in-person meetings and thus, by extension, the communication overhead in a team.

I want to close with an analogy that I think captures the problem of software estimation quite well. With estimation, like much else in the field of project management, the devil is in the details. If I were to ask you to estimate the time to get from San Francisco to L.A., your response would likely be something like 2 hours, on a flight. If I were to ask you how long it would take if we were to drive instead, you would then update your estimate to something like 6 hours, assuming everything goes smoothly. Obviously, you can’t really plan every minute of the journey. Every unexpected detour will compound and degrade the quality of your estimates.

Software development estimates are no different. They get better with intuition and more information, but can never be quite perfect. It is nearly impossible to lay out a neat little roadmap of all the steps that need to be completed and assume that everything will go just as planned. The best we can do is use our judgement, collaborate with others, and be aware of some of the blind spots we may have. Hopefully this post helped with some of that :-)

If you found this post useful, be sure to leave some 👏👏👏. Let me know in the comments below about other topics you’d like to see discussed. Feel free to connect with me on LinkedIn.