Return-Driven Decision-Making on Software Projects

What is Return-Driven Decision-Making?

Doing a project is like making an investment. A company does a project by investing engineering resources, hardware resources, operational resources, etc., in hope of (eventually) making more money than they initially invested.

As a project owner, managing your projects is just like managing your stocks. Before buying a stock (initializing a project), you need to decide if the stock is worth buying (if the project is worth doing). You only buy that stock if you believe that stock will have a positive return in the future, and the return should be large enough to worth the effort. You may plan to hold the stock for a period of time (plan the project duration), but if you estimate that you will lose money if you keep holding it, you will immediately sell the stock (abandon project) even though it hasn’t reached your financial goal. If you estimate that stock will give you more return in the future, you might want to hold the stock longer than you planned or to increase the investment (add more features to that project). Every decision stock investment is about the financial return.

Software project decisions can be made in a similar way. Essentially, you want to maximize the return, which is the difference between the gain (how much you have made) and the cost (how much you have invested):

return = gain — cost

Value metrics

Unlike the stock market, where you can always measure your return in dollars, measuring return in a software project is more complicated, because the gain and the cost are measured in different value metrics. For example, a project whose gain is to reduce p99 latency by 50% does not have same value metric as its engineering cost (e.g. one SDE working one month to release this project).

Here are some typical value metrics:

• Dollar, such as sales this project generates, the hardware cost, etc.
• Engineering cost, such as how many SDEs, how long they work on this project
• Latency
• Throughput
• Scalability
• Usability, such as saving one mouse click to login
• Complexity
• Customer trust

The challenge is that to be able to compare different value metrics, you need to project non-dollar value metrics to dollar value metric. For instance, figure out much 1ms latency is worth in dollars in your service. Some conversions are easy: you always know the SDE’s salary and the hardware cost; Some conversions are very difficult: how much does 1-mouse-click worth in dollars? Many companies use A/B tests to measure this.

Even though you can’t exactly project all non-dollar value metrics to dollar value metric, you at least should have a reasonable value range.

Maximize the return

Because return is the difference between gain and cost, to maximize the return:

• When looking at a single project, you need to maximize the gain and minimize the cost.
• When comparing the different projects, you need to compare the return. Neither gain nor cost alone can help you make right decisions.

There are two common pitfalls when they make decisions in software projects:

1. They only think about gain, don’t think about cost. They usually rush to a project without thinking about the engineering cost and opportunity cost. The right thing to do is to estimate both gain and cost, calculate the return. Among all candidate projects, only pick up a project that has the highest return.
2. They only think about the cost, and ignore the gain. This pitfall could manifest in this situation: preferring “a feature takes 2 days to implement can make the operational load a little better” to “a feature takes 20 days to implement can make the operational load a lot better”. When making this decision without estimating the dollar value of “a little better” and “a lot better”, a wrong decision can be made.

Return-Driven Approach in Software Project Life-Cycle

Regularly use return-driven decision-making approach

In a project life-cycle, project return should be constantly re-evaluated, just like investors periodically evaluate the stock and estimate the return based on the new information.

While many projects conduct thorough initial scoping and planning, not many projects (especially smaller projects that span less than 6 months) actually do a regular re-estimating on return, while they regularly adjust the release dates. For those projects, my suggestion is that when adjusting the release dates, also re-estimate the project’s return.

Project Initialization

Applying return-driven approach in project initialization phase, you need to answer this question: among all other candidate projects, can this project bring the highest return when it’s delivered?

Project scoping tries to answer this question by gathering information to estimate the gain and the cost. Talking to customers to understand their use case is a typical way to estimate the gain. Estimate the engineering cost and hardware cost is a typical way to estimate the cost.

When making a decision on whether to pick up the project, be aware of the pitfalls mentioned earlier: either gain and cost along can help you make the right decision, only return can.

Project Design/Implementation

If I want to see if a team can make rational decisions, one metric I would look at is how many projects have been canceled/abandoned in the design and implementation phases because the team does not believe this project could bring positive return (assuming the team already estimated that this project could have positive return in project initialization phase).

I find that developers rarely re-estimate the return after a project has been officially kicked off, while the leadership does that, but less frequently. It’s very common that, in the agile development, developers re-estimate the points they need to spend on different tasks in design and implementation phases. Often, they also need to adjust the project’s release date in the implementation phase. Nevertheless, it’s uncommon that developers frequently re-estimate the project’s return based on the new information they have learned in the design and implementation phases.

One of the famous fallacies in economics is sunk cost fallacy. While it happens to many inexperienced stock traders, I wonder if it happens more often to software projects, because developers can easily build personal bonds to projects when they invest a large amount of time and energy. I seriously doubt that a team working on a project for 3 years would have the guts to abandon this project if they found out that the future return will not be positive.

Even initially a project’s estimated return is positive, it could be negative later, when:

• During the implementation, the team finds out that the cost of running this feature is much higher than they expected, which will not bring a positive return.
• Their project has been delayed many times, so their customers have used other solutions.
• They find out that the problem they try to solve does not exist anymore because other dependency systems or user behavior have changed.
• Completing this project next month will have less return than completing it 2 months later because a complex component can be delegated to a new service, which will be released next month.

More situations can make the project’s return negative. Unfortunately, those situations are not or could not be anticipated in the project initialization phase. At this point, the team should evaluate the cost of abandoning this project and use return-driven approach to decide if it would be better to abandon this project.

Feature Deprecation

After a project is completed, the team needs to maintain features delivered by this project. Depends on the design, implementation, architecture, or upstream/downstream services, a feature’s gain and cost will change constantly. Thus, a feature will generate either positive or negative return as the time goes by.

A regular return analysis should also be done after a project is completed. A return could be negative in either of two situations:

• Operational cost is increasing. The team needs to spend lots of engineering resources to maintain features released by this project.
• Gain is diminishing. It may be caused by users moving to a cheaper and/or more efficient solution.

When the project’s return is negative and the analysis shows that it’s unlikely to be positive in the future, the team can:

• Propose a deprecation project to eliminate its cost (as well as its gain).
• Propose new projects to increase the gain.

It’s worth noting that deprecating a feature is not free. As any regular features, return-driven decision-making approach can also be applied to feature deprecation project. There are two types of cost in a feature deprecation project:

• Cost of off-boarding existing customers. This means that your gain will be less. It also means that during the feature deprecation, your return will diminish more rapidly until you completely release all the hardware resources used by this feature.
• Engineering cost for developing this project.

Sometimes the cost of deprecating the feature is so high that results a huge negative return. In this case, stopping on-boarding new customers and continuing supporting existing customers to confine the negative return is a better decision than deprecating this feature. This is especially true if you consider that hardware cost is always decreasing and the engineering cost is always increasing.

Apply Return-Driven Decision-Making Approach to Multiple Projects

Return-driven decision-making approach can also be applied to multiple related projects (e.g. projects for one service) together. It’s not uncommon that a single project could have a negative return, but it will generate positive return when being combined with other projects later. When being applied to multiple related projects, return-driven approach can:

• Mitigate the risk of an individual project.
• Provide a holistic view of the return of the service as a whole.

When you zoom out to estimate the return of multiple projects together, then zoom into a specific project, you will be able to estimate the return that this project brings to its related projects, in addition to the return generated by this project alone. It is a risk if you look at one project, but it could be a win for the service as a whole.

Limitation of Return-Driven Approach

Return-driven approach is not useful when the gain and/or the cost cannot be reasonably estimated. This could happen in situations such as proposing long-term visions for a team or exploring new business domain, or new market.

The gain in those situations cannot be reasonably estimated because:

• long-term visions do not always have concrete customer use cases; Long-term visions usually focus on what the team wants to do for their customers in the future, not necessarily on what customers want right now.
• new business domain or new market do not always provide enough data for estimating the gain. A very small market may no longer exist in next 3 years, but also may become a huge one.

The cost in those situations cannot be reasonably estimated because cost, in most cases, depends on how to implement those visions and how to explore new business domain and new market.