Hofstadter’s Law: It always takes longer than you expect, even when you take into account Hofstadter’s Law. — Douglas Hofstadter
A Product Manager friend of mine recently told me about a problem she was having: “Software engineers can never estimate how long their projects will take. What can I do?” Two CEOs recently told me the same thing.
We engineers have all witnessed this too. I once saw a project estimated at two days take four months. In that case even the “just double it” heuristic would still be off by an order of magnitude. This can have real implications for the business. I’ve seen a whole company move mountains for a launch event that had to be pushed out months.
At a high level the problem is a difference between what engineers mean when we estimate time and what PMs, managers, PR, and really everybody else mean. Most engineers instinctively think about the minimum time to write a working prototype if everything goes pretty much as planned. But those blocked downstream want to know when the project will be ready for launch — and that’s a totally different story.
For engineers, mastering estimation is a lifelong journey. Neglecting it will plague you and everyone you directly or indirectly interface with. Mastering estimation will set you apart and your colleagues will associate you with professionalism, stability, and quality work.
Why we need to estimate
Let me start by answering the question I most often get from engineers: “Why bother?” Many engineers complain (correctly) that it’s an overhead cost. “I’ll finish sooner if I just power through on it until it’s done!”
There are two main reasons: external dependencies and prioritization.
External dependencies
Nothing impactful operates in a vacuum. Projects often have external dependencies like coordination with non-engineering teams (comms, finance, PR, customer support), other engineering teams, or even end-users themselves. It’s typically the job of the manager, PM, or CEO to coordinate with these external dependencies. That means that the one who is best qualified to make a time estimate (the engineer) isn’t the one who needs it most. This asymmetry leads to a fundamental tension.
Prioritization
Time estimates are also key for prioritizing work. “Bang for the buck” is an important metric in engineering and there’s no “buck” without real estimation. Even if the feature you’re working on is the most awesome thing in the world, if you take the time to do a full estimate, you might realize it will take way too long to finish.
Say you’re working on a project that will make the website 50% faster but in the same amount of time you could have finished two projects that will each make the website 40% faster. If you don’t take time to do an initial estimate, you’ll never know that you could have ended up with a much faster website!
Time estimation 101
Now that we all agree that time estimation is necessary the vast majority of the time, let’s talk about techniques.
We underestimate time because we think “How long would it take me to write a basic version of this?”
But shipping is much more than a basic version. You will need to account for the time it takes you to write, test, debug, and polish. Don’t forget the time you’ll be in meetings, interviews, doing code reviews, sending emails, etc.
Another reason we underestimate is that we almost always encounter “unknown unknowns” in the coding process itself. And those are impossible to anticipate fully and account for. Maybe your IDE will get an update that breaks your project and you’ll burn a day fixing it. There’s no way your estimate could have taken that into account.
But we can still do much better than our initial instincts. Here’s what I do:
Step 1: Make a technical plan
You should already have a technical plan or design doc ironed out for any nontrivial project before you begin. You use this to let others know what you’re doing and get feedback. The technical plan is the ideal place to start the time estimate. As you work through the technical details, you’ll already magically be improving your estimate as you uncover unknown unknowns. Maybe you’ll realize that you probably will need to upgrade to a new version of a library that you’re using and that could add a day. You might even realize the library you were planning to use doesn’t actually exist and you’ll need to write it.
Granularity is important here. If any step feels murky or vague you’re either hand-waving (and should learn more) or you need to break it down into smaller steps. At the same time if a step is too fine-grained it might be brittle enough to invalidate the whole plan in practice.
For a good guide on what sort of thinking should go into your technical plan, check out this article by Alicia Chen. One key point is to iron out any potential ambiguities with the PM or other stakeholders so that you don’t end up building the wrong thing and having to start over.
Step 2: Add a time estimate to each step
Estimate how long each step in your technical plan will take to implement. This will often involve research into the details (“is there already a library to do this or not?”). Depending on the nature of the project, throwing together a simple prototype might help reveal a lot of potential future pain points.
Step 3: Add in a bunch of extra time
Now that you have a barebones of your estimate, there are all those things we mentioned earlier to account for.
- Debugging as you go: Bugs always come up. A lot of this depends on your experience with a specific codebase and the codebase’s maturity.
- Meetings, interviews, vacations, etc: You probably won’t be at your desk coding the whole time. How many hours will you REALLY have to code? You should at least look at your calendar when estimating.
- Final testing and bug-bashing: You should generally be writing tests as you go, but a lot of teams need to do an extra round of polish work or integration testing before launch. Give ample budget for it in your estimates. If you’re doing a staged rollout, the initial 1% rollout will probably reveal bugs that need fixing. Account for that.
- Code review: How many rounds are typical for you in this codebase? How long do they usually take? Be sure to verify an ample supply of reviewers (and maybe check their calendars). If it’s the kind of project where there’s only one possible reviewer you should solicit a commitment in advance and ask them for a backup in case they’re on vacation or way too busy at a critical point.
Once you start adding in all of these costs to shipping, you’ll start to see your time estimates match up a lot better with when your projects actually launch. Yes, they’ll be longer. Yes, you might feel pressured to shorten them. But when people figure out they can depend on you they’ll come to appreciate your estimates.
Step 4: Review your estimate after you’ve launched
Yes, it sounds like a pain to go back to your time estimate after you’ve finished a project and review what you’ve done. But this review is how you learn and get better next time.
What ended up taking a different amount of time from expected? If integration testing took twice as long as you thought, write that down and leave more time for it next time. Or try to improve your integration testing system.
You’ll definitely see your estimates improve with time. You might even come up with some great insights here that will help your whole team.
In the end, it’s all about communication
Communicate your timetable and changes to it early and often. If you let your manager know a month before launch that there’s a new security bug in the library you were using and you’ll have to start over from scratch, they’ll in turn have time to notify PR, finance, or users that there’ll be a delay.
Communication to relevant parties also lets them give you important information that can affect your estimate. A designer might say, “Oh, if that fancy animation is going to take a whole week we can just cut it completely.” A PM might add, “This is just a prototype to experiment on in user studies. We don’t need to do much bug bashing for this iteration.” A manager might say, “You’re spending half your time in meetings? Let me fix that!”
For engineers, don’t give in to pressure to report a shorter time than is realistic to appease the higher-ups. It is more professional to be honest about your estimates and how they’re changing.
For everybody else involved, respect that estimation is hard and that it’s going to be a process. You can only cut down time estimates by sitting down and removing features or stages that aren’t actually going to be needed for launch. You can’t cut it down by nagging.
We’re never going to be able to perfectly estimate time of a project. The only way around this is an open communication, compassion, and relentless prioritization.