Why are engineers so bad at estimation?

Estimating engineering tasks is hard. It’s a skill that can be improved with practice, but even experienced engineers are often way off base. Let’s look at some of the reasons why and discuss some options for dealing with this inevitability in your company.

The requirements were not fully spec’ed

This is probably the most common reason that a project can take much longer than expected. If the engineer doesn’t know the details of what is required of a task, then there is no way their estimate hit the mark. (Most often, they will underestimate, as they won’t factor in the details they didn’t know about.) Details that seem small to you might actually be the bulk of the work for your engineering team. UX is often the culprit: the complexity of implementing a visual interface to spec can vary very widely depending on the design. A classic (but dated) example is rounded corners: in the past, it was particularly painful for web engineers to add rounded corners to images or tables and they would bicker with designers about whether they were really necessary. I try as hard has possible not to start engineering work on a project until the mockups are 100% complete. Mockups also help engineers (and everyone, really) better understand what’s involved in a change that might be missed in a written spec.

Then there are changes in scope. While these changes may be unavoidable — business environments change, and you sometimes don’t realize you need something until you try out a working prototype — they always decrease efficiency and make projects take longer than expected. Some of the work already done when the scope change comes in may have to be thrown away or significantly updated, not to mention the new work related to the new requirement. Keeping projects small and delivering them piecemeal helps with this, but it’s always going to be more efficient for your engineers to minimize scope changes.

Engineers may estimate the best case scenario

An author wouldn’t predict the amount of time it would take to write a book by multiplying the number words per minute they could type by the expected number of words in the book (I could write a 100,000 word novel in just 21 hours!) But this is often how junior engineers reason: they neglect to consider the time it will take to think. Software development takes brainpower to architect the best solution; that’s why you pay us the big bucks. Further, engineers may neglect to consider testing, bug fixing, or any refactoring that may be necessary to complete the change.

The most time consuming part of a task may be something that was totally unanticipated. Perhaps a new library that the engineer was planning to use to accomplish the task didn’t work quite as expected or a gnarly bug took days to fix. We have all had experiences where we embarked on a seemingly small task, but then ended up bashing our heads for days on something that we thought would be trivial.

Engineers have other things on their plates

A bad way to use your engineers’ estimates is to ask how many hours it will take to do something, then divide that by eight to determine how much work they will complete per day. It’s pretty much unheard of for an engineer to sit down at their desk at 9am every day and code nonstop until 5pm. Setting aside some engineers’ creative work hours, there are lots and lots of other things that get in the way of pure productive bliss. Some may be looked down upon — admit it: you check Facebook and your personal email occasionally during work hours too — but some are just part of the job. Meetings, work email, interviews, Slack, etc. are all things you want them to be doing; or, at least, the work culture your company has defined demands it.

Then there are miscellaneous technical tasks: that small feature you wanted to squeeze into their current workload, the high priority bug fix, the production outage — those all take time. Worse, last minute changes are especially detrimental to productivity because they take engineers out of “the zone” when they are most productive. Engineers use the term “context switching” to capture this. The term originated as a technical concept (operating systems take a similar efficiency hit when multitasking), but it applies well to the loss of productivity. The idea is that every time an engineer has to switch tasks, they lose their thought process and have to acquaint themselves with the new task at hand, thereby reducing efficiency and resulting in the overall cost being more than the two tasks individually.

Debugging tasks are basically impossible to estimate

It is particularly challenging to estimate the time it will take to fix bugs. The vast majority of the time spent fixing bugs is in diagnosing the problem. Typically, after some period of debugging, the engineer has an “a-ha!” moment and can fix the problem with just a line or two of code. The debugging process itself is the long pull, and there’s no way to know in advance how long that will take. If they knew exactly what the problem was, they wouldn’t have introduced the bug in the first place.

Pressure from you

Sometimes, providing estimates feels more like a hostage negotiation than a candid exchange of knowledge based on technical expertise. Phrases like “this doesn’t seem like it should take 2 weeks” or “can’t you just get it done in one week?” undermine trust. Presumably, you’re asking your engineer to provide estimates because you respect their expertise. Depending on the nature of your relationship, the engineer might push back or might cave to your pressure, which is more likely to result in technical debt than more efficient engineers.

Rather than demanding that your engineer change her estimate, seek to understand why it’s higher than you expect. It’s quite common for the long pull to be something you don’t actually care too much about. Perhaps removing an animation or using a third party for some component could drastically reduce the amount of work involved for the engineering team. Especially when working with startups, I try my best to understand the real business requirements and apply my technical knowledge to reduce new products or features to their core MVP. The 80/20 rule — 80% of the benefit comes from 20% of the work — very much applies to engineering.

But my business can’t function without planning! What should I do?

The classic answer is “Agile” (imagine harps playing while reading that word). Agile is a buzzword that encompasses a variety of software development principles that are adaptable to change. While there are several concrete implementations of Agile philosophy (such as Scrum and Extreme Programming), the common theme is that software development is collaborative, flexible and iterative. Detailed product specs for multi-month projects are deemphasized in favor of smaller units of work and working in close collaboration with stakeholders (product managers, customers, you, etc.)

But wait! You said earlier that I need to create detailed specs in order to maximize the productivity of my engineering team. I stand by this: a thorough spec that is impervious to change will optimize your engineers’ efficiency. Agile methodologies acknowledge that this holy grail is unrealistic: there is no way for the writer of a spec to anticipate everything, and it’s better to minimize the negative impact of change than to pretend it won’t happen.

Instead of detailed specs, Agile methodologies encourage an iterative process of small changes rather than a huge spec that outlines several months worth of work. It encourages you to empower your development team to take ownership of the product and make small implementation decisions by themselves. If you disagree with their micro-decisions, their frequent check-ins and demos ensure that they don’t go too far down the wrong path before course-correcting. Engineers will likely implement the simplest solution first, then you can decide together whether the more time consuming tweaks are more important than other tasks coming down the pipeline.

One cornerstone of Agile methodology is the concept of using relative “points” rather than “hours” or “days” to estimate tasks. Your team can then establish their “velocity” by calculating how many points they complete in a given time period and use that to decide how many points to take on in the future. Using an abstract notion for estimation (eg. “this task is 3 points”) offers a few benefits. First, we tend to be better at estimating things relative to each other than absolutely. That is, we tend to be more accurate when we say “this will take twice as long as that” than “this will take 10 hours.” Second, it solves the miscellaneous distraction issue, at least as it pertains to making plans based on estimates. Assuming distractions and miscellaneous tasks are relatively constant, you can be fairly confident that the team that completed 20 points last week (based on their own point estimations) will be able to complete about the same number this week. Finally, using points reduces the temptation to bicker about estimates. Nobody is going to get bent out of shape by saying a task is 13 points, whereas an estimate of 13 hours may be subject to more scrutiny.

Regardless of how fully you and your team embrace Agile concepts, frequent communication between your engineering team and stakeholders is key to keeping them efficient and keeping tabs on the progress of development. By checking in frequently and directly seeing their progress as working code, you can jointly make decisions about how to proceed. If a project isn’t as far along as you expected it to be, you can decide together whether to accept the altered timeline or drop some of the requirements to reign in the end date.

Hope you found this post useful! Don’t forget to follow me here on Medium, my blog WTF Is My Engineer Talking About, Facebook, Twitter, or LinkedIn. And please send feedback and topic suggestions via e-mail.