A developer’s tale of deadlines and fantasy time estimates

You were asked to write a program, and during the discovery phase of the discussions, you were asked how long it was going to take. The spotlight was on you and everyone turned to face you. On their faces you could see the real question they were asking… How long are you going to set us behind our actual target? Cause let’s face it, that is what they really want to know. How long do they have to wait before they can use and advertise that new feature to the market.

You start to feel the pressure that everyone is suddenly counting on you. Maybe you are the new developer on the team, the last thing you want to do is to let everyone down. Without any real research or details, you make a quick mental calculation and ballpark a time frame, based on what you know so far.

“It could take me about 6 weeks, give or take”, you say, all the while hoping they note you said “could” to imply speculation and not a fixed possibility.

“Okay! If you could bring that down to 4 weeks that would be fantastic yeah?” is what you hear.

Everyone is excited. Your manager is brimming as they note the time frame down and base plans around that schedule. They are expecting a working program in 4 weeks. You look around at everyone making plans around that time estimate, and that’s when it sets in, a nagging uneasiness at the back of your mind; you didn’t factor all the things that could go wrong and will go wrong. A nagging thought starts to settle in, there is something you definitely forgot to account for. Anyway, what’s done is done, you say to yourself. You decide you will do the best you can with the time you have and hope everything goes as close to the plan as possible. Let the journey begin.

During week 1, you finally receive the design documentation and program specifications. As you read the specifications, you notice some additional bullet points that were not mentioned in the initial discussion. You confirm with your manager if this is real or enhancement requests after the initial release. To your worst expectation, it is real and part of the initial requirements. You should have known not everything was mentioned in the discovery phase. You compare what you thought you were building with what you shall be building with the new design specifications. Somewhere by the end of the week 2, you have managed to jot down a few options on what possible approaches you could take to tackle the problem. You have mentally weighed their observable pros and cons (yes, observable because sometimes appearances are deceiving) and are close to selecting a single approach. You scribble a quick flow chart and theorize a couple of tests that your program must pass and those it must fail, basically define the boundaries of your testing conditions.

By the beginning of week 3, you have a clear direction to follow, you are aware you have only 2 weeks left but you feel confident enough since your action plan is already set, all you have to do is build the parts. And so you begin. Everything is going well and midway to week 3,  you have begun testing a few of the components you have already built as you integrate them into the main program. Sometime during the middle of the third week, at the start of the fourth day, a critical bug gets reported on a different project and it requires your immediate action. You question whether it is really critical or just a slight exaggeration. You get confirmation from your manager confirming it is critical, if you could get right on it. You really don’t have time for this but oh well you must take a look at it. What’s the worst that could happen you wonder.  Naturally, you take on the task and sideline your current project. Suddenly, your co-workers are signing off and to your shock, it's already the end of the day. You spent the entire day working on the bug, and to your dismay it's the end of the day, you haven’t fixed the bug nor made any progress on the project. You start to feel just a little nervous. Next day, you are still working on fixing the bug, this time more voraciously, you have to get this done because all this time is costing your current project. Luckily by the end of the day, you have a working bug fix. You aren’t too happy with it because you haven’t had time to thoroughly test it, but then again you are way off your schedule, it's already midday.

By week 4, you can finally get back to your project. You start to get a feeling of urgency creeping up on you. This is the deadline week. You psych yourself up, if you focus on this and only this, you could pull it off by a hair. Just as you start working on the project, you get called into a sync meeting, everyone is giving updates and all other parties want updates so they can revise the time frame. You are shocked; you completely forgot about this meeting. That takes off another 40 minutes at least from your time. You can feel the need for an extension, however, you haven’t figured out how to ask for one without seeming like you are just asking for the sake. You quickly put together a few slides with your flow chart showing what you have done so far and what you need to do. If you are going to get an extension, you will need everyone invested in understanding why it is important you get that extension. 

During the meeting you discuss what you managed to get done, the problems you encountered and how you intend to fix them, for those you still haven’t worked on, you discuss the strategies you intend to take when implementing them. Finally, with all the cards on the table, you make a request for an additional couple of days to complete your part in the project. Your manager eyes you curiously, then says okay, you get your extra week. You remain level headed, but inside you jump in relief. You didn’t need a whole week, but with a whole extra week, you could actually do more thorough testing. As you work to complete the project, you make a mental note to next time, not give time estimates, or better yet, ask for time to think about how long you actually think it will take. However, deep down, you know you will be doing this fantasy time estimate all over again on the next project, because for some reason, that is just how it goes.

So why is this all just fantasy?

Software development is subjective to a lot of the road blocks encountered in any creative process

I have come to realize that writing software is a lot more of a creative process than people realize. The programming languages we choose, frameworks, libraries etc. are all technical, but how we implement them, integrate them or manipulate them to solve a problem is inherently a creative process at the mercy of the person tasked with solving the problem. It is the reason multiple solutions can exist for the same problem. Everyone can have the same tools, but the masterpiece each produces is different and subjective to their creative process.

We go through the same motions as any creative writing. We have times when the plan is clear and times when we stare at our editor not knowing what to start with or what to write next. We design how everything is connected and where they are connected, and keep revising these decisions as requirements change. We consider each user individually and at the same time we consider the collective users and how we might meet their needs. We write a bunch of code one day, then delete all of it another day cause it just doesn’t sit right. We wake up some days confident with what we have done so far and some days we second guess every decision we have made thus far. This goes on until we have a final product. It's a whirlwind of ups and downs.

Time estimates in a creative process are akin to a sort of fantasy time. We hope it will take X amount of time, but at the end of the day, we truly do not know, it is just a hope based on similar situations or experiences.

There is no real one size fits all solution

The assumption behind time estimates is that problems that appear similar and are of similar complexity should by extension take a similar amount of time to solve. If you really think about it, if it was the same problem, all one would need to do is plug the same solution. Similar is not the same. Therefore however “similar” problems appear, they are inherently not the same and therefore require modifications to make them work. The time associated with these modifications is not always known or constant as each problem is a new venture.

You probably have a lot of tasks you are working on

Ideally, when you initially estimated the time it would take, you did not factor in external environments such as how the other projects you were working on would still need your attention.  When asked for your time estimate, you probably isolated that project from all the rest. What you really meant was “Without other commitments and in a perfectly isolated environment, it could take me X amount of time if all other factors are held constant”. By factors you meant, if program specifications do not change, if no additional requests are made and if you do not have to share your time with other projects.

Changing program specifications midway does not affect time in any predictable fashion

There is an assumption that removing specifications directly means the software can be completed in less time. While this might be true in certain scenarios, I have yet to discover this as truth. Ironically, what I have experienced is adding or removing specifications midway just affects time estimates in the same way; they no longer apply. It does not mean you will finish the project faster. All it means is you have to revise your software and consider how this change will affect the work you have already done thus far. It also means you have to consider what you would need to add or remove to meet the new program requirements.

Things will go wrong and sometimes there is no quantifiable time taken to fix them

Some time during the process, some unforeseen problems may occur and it is entirely possible, it could take days or weeks to figure out how to solve those problems. Problems may come from your computer, your chosen language, framework, dev environment, latest updates released by one of your dependencies, inability to integrate third-party tools properly etc. It could even come from the solution you chose to implement. Turns out, maybe the testing phase revealed it is not as efficient as initially estimated and is therefore not optimal and now you are back to the drawing board 9 days before the deadline. Factoring for potential errors is difficult as you cannot foretell what problems may occur and how long it will take to solve them.

There is a lot of interdependence within the system

When collaborating to write software in a team, it is common to split the tasks between team members. Ideally, each of the tasks can be completed independently, however, in some cases, they could depend on each other. If one part lags, it drags the dependent part with it. Typically, to curb this, it is ideal to simulate the input and write one’s section accordingly as you wait for the actual input. Sometimes, this approach works but in some cases it may be harder to do. For example, it can be difficult to simulate images that require subsequent image processing to remove defects or unwanted artifacts without having an idea what the defects could be. Therefore a delay in completing the first part that supplies input to a second part, may delay the second part as well.

In a team, collaboration is at the heart of writing chunks of software. You may be struggling to fix something and subsequently affect your teammates ability to work on their part, because it relies on you finishing yours. Thereby not only affecting your time schedule but theirs too putting pressure on both of you. Therefore having proper estimates that account for delays on your side, could help your teammates estimate how they are also affected and can adjust the schedule appropriately without allowing the stress to beat the deadline fall on them.

You aren’t good at estimation

It’s easy when others place the time estimate on you and sure you can blame them, but sometimes, even your own time estimations just aren’t good enough. You thought it would take an hour, it took 2 hours. You thought it would take 2 weeks, you were given 1 week, you missed the deadline and were given an extra week and you still missed the deadline. Therefore, your initial estimation was still wrong. What are we without idealistic human optimism? It is easy to think of oneself as being able to power through the mental drain writing software takes, however, more often than not, mental drain comes coupled with physical drain and the time taken to rest and power up in between is often overlooked. Small things often add up to big things. Taking time to rest, brainstorm every once in a while, are tasks that are difficult to accurately quantify but do matter and can take up quite some time. They may seem trivial when ballparking an overall time estimate, but they significantly affect the time taken to write software.

So why do we still need real time estimates?

If time is all a fantasy why do we still need and rely on it? As much as software development is a creative process, it’s aim is to have real-time use.

Your users have a real-time use for it

Imagine if you sat down at a restaurant for lunch and planned to order the special. However, when you asked the waiter how long it would take, they told you they had no clue, it would be ready when it was ready. I am pretty sure you would order something else. How do you plan the rest of your day around a lunch that has an unknown time factor in it. I mean you could gamble and order it and best case scenario you get your lunch in 15 to 30 minutes, or find yourself hours later still waiting for lunch.

Need has a time factor to it. Something needed in the next 3 months does not apply if it arrives in 6 months. Therefore, you may have to plan what is feasible around your user’s needs. If you are organizing a marathon and need an app made to track runner’s location, pace, vitals, speed and direct them along the proper route, there is absolutely no use in getting the app long after the marathon is over. It no longer solves the problem. Giving a time estimate to how much time would be needed to create the app can give stakeholders proper time to plan around the deadline or even start the project earlier than intended to meet the user's needs. In the marathon scenario, if it is too late to create an app to meet that marathon, maybe for the next marathon, the project can be started well before the marathon’s date with the time estimated.

Deadlines can help one stay focused

I tend to procrastinate sometimes and when it comes to my side projects, they can suffer especially when my interest has faltered, or when I have become deeply obsessed with a different project, and there is no fixed deadline. Time estimates help me stay focused on a specific task for a certain amount of time. Even though I may procrastinate within the set time frame, at the back of my mind, I am aware there is a project that needs to be completed and I will carve out time to get it done.

Time estimates do aid in overall planning

I don’t think people ask for time estimates because they enjoy putting people under pressure, if they do, you should probably reconsider your work environment. Time estimates help others plan their part of the project around a given time frame. A lot of software is integrated or built to work hand in hand. You can’t launch a product if part of the product is missing a vital area. Since the entire system is the sum of its parts, how long it will take is affected by the time estimates of each part. With near approximate time estimates, the project manager can decide which parts can be created in parallel, which parts can only start once a different section is complete and overall how this affects when they can complete the project.

We tend to understand the reason for time estimates and why we sometimes fall short of how long we think something will take. So the next step would be to re-evaluate how we estimate software design.

So how can I get better at estimating?

I still catch my breath when someone asks me, “How long will this take?” Why does this question cause me so much anxiety? Well, because at the back of my mind, I am aware of all the bits that go into time-estimates as well as all those we typically overlook. However, with time I have come up with a bunch of steps I take to break down how long I think a project will need. I haven’t nailed it down to the second, however, with time I have gotten better at giving better approximations on how long I think I will take on a project.

Give a rough estimate and ask for time to break down the problem and come back with a better estimate

Nowadays, I give an initial rough estimation based on the mental gymnastics I can perform at that time, however, I also ask for some time to come up with a better time estimate outside of the meeting environment. Doing this often allows me to sit with the scope of the project, break it down into chunks and identify what I think each chunk will take. Once that is done, to each chunk I add various miscellaneous time estimates, “Exploratory time”, “Bug fix time” and “Pause time”. These are not industry terms just personal terms.

“Exploratory time” is generally the time I estimate I will take to consider my options in determining how to implement that particular chunk. Great thing about writing software to solve problems is there can be multiple solutions to one problem. Ironically, this can also sometimes be a problem in itself. Too many options can lead to analysis paralysis when trying to determine which approach one should use given their particular problem. Personally, when choosing options, I do take time to consider how each option could solve the problem, their flexibility, advantages, disadvantages, future maintainability and general interdependence. I also consider, if I choose to change my approach later on, what will be the cost to switch to a different solution and how difficult will that be? I have found that adding an exploratory time also allows me to research without feeling like I am costing the overall time, because I accounted for time to explore my options in depth.

“Bug fix time”, is generally the time I estimate it will take me to find and fix bugs in a particular chunk. When writing software that is similar to something you have implemented previously, this time estimate might not be necessary, however, for newer programs, it might just be necessary. I find having this time is a safety net for when I get stuck on a bug that I am unsure how to solve. It isn’t easy to estimate how long it will take to fix a bug, but estimating there will be no bugs at all is worse. When bugs do pop up, urgency and subsequently stress set in because you had not budgeted for time to fix bugs. This has happened to me in the past, and I hated it. However, expecting things to go wrong and having time set aside to solve them can leave you feeling calm and still on time in the grand scheme of things.

“Pause time” is generally a nice way of saying break time. Time to step away from the software, take a break and refresh. Each chunk gets its own pause time because not all chunks are the same, thus their mental and physical drain can be different. The break I need after fighting with css may not be the same break I need after writing tests.

Generally, consider your habits when writing programs, how long did those small things you overlook take? Find a way to term them and add them to your general time estimates.

Estimate sections you are familiar with differently from those you are unfamiliar with

Familiar things are easier to estimate because we have done them 10, 100 or even 1000 times. We know the general flow and how long it takes us to work on them. However, even things that seem similar are not always entirely similar in every aspect. Nonetheless, even minor variations are easy to handle because they only deviate so slightly. Therefore, their estimates can be close enough to problems previously encountered.

Things we are unfamiliar with are harder to estimate because we have never done them before. Maybe we have heard of them and know the general approach, but we have never actually sat down and implemented them. So how do we estimate something we have never done before? One approach would be to use estimates based on other’s experience. When we do not know how long something will take, we could either make a speculative guess, or rely on those with experience on the matter and ballpark our estimates based on theirs. It is not ideal, but it is better than magically producing a number. However, there are a lot of unknowns that come with using other people’s estimates on how long a task should take. For one, time estimates on known matters are subjective and can be influenced by experience. Someone who has designed and written hundreds of application code that queries databases may take only X amount of time to implement something similar when compared with someone who has never done it before. Knowledge in a certain domain also influences how long one takes to write code in that domain. We are all different.

Personally, when it comes to estimating the time I will take to work on something I have never done before, I tend to take a senior’s time estimate then double it. By doubling it, I can have time to implement it twice in theory. Once to get it done, and I use the extra time to review my understanding of the task, setting myself up for the next time I may have to implement a similar task.

Remember to take your personal habits into account

People are different and have different habits. How we tackle problems, how we brainstorm and eventually how we assess and handle situations is also different. Some people prefer to tackle the same problem for hours on end then take breaks, while others prefer to concentrate for a couple of minutes with short breaks in between. Some would prefer to work on mentally tasking projects during the evening while others prefer working on such tasks in the morning hours of the day. Some people can’t work on an empty stomach while others can’t work on a full stomach. Nonetheless, whatever your preference is, consider how your habits affect your performance and take them into account when estimating. Will you need more breaks in between or not. Do you need to devote ample time to doing research or not?

Keep your team informed, and if revisions are necessary, consult them

All in all whatever happens, always keep your team in the know. I do realize this is not a direct way of getting better at giving time estimates, but constantly communicating with your team about your efforts and where you are in the timeline can allow you and your team to revise the timeline and everyone that is affected is informed well ahead of time. Remember writing software and launching a product is a team effort, you may not think they are affected by revisions to your timeline, but they probably are.

Lastly, learn from your own estimations

Every time you work on something, note how long you thought it would take, and how long it actually took. Then compare how far off your estimation was. Were there outside factors that caused this value to deviate exceptionally, and if there were what were those factors. With time and learning from your own experience, you should get better at estimating.

By having close to accurate time estimates, you will realize you may finish projects earlier than expected if everything goes right. When things do go wrong, you still have sometime to fix them without affecting the overall project deadline.

Hope you enjoyed the post, let me know what you think in the comments section or reach on me Twitter, @loetcodes.