Feature Prioritization: Why It's Difficult, and How to Do it Better

When we're prioritizing, we're trying to answer the question "What, among all the stories we could work on right now, is the best possible use of our time?" The obvious answer is to say "The story that adds most value to our users". This answer only leads to more questions such as:

• 🤔 What do we mean by "adding value"?
• 🤔 How certain are we about our assessment?

Managers have to answer these questions for themselves. And then, they have to open up their conclusions to debate among their stakeholders. A clear and concise prioritization strategy is essential to ensure alignment among all teams involved.

This strategy needs to be clear so that it's easily understood by all the various stakeholders who may or may not have the same amount of context. It needs to be concise so that the logic can be easily propagated among colleagues and alignment can be maintained.

In this post, we go through three barriers that prevent us from finding a clear and concise way of prioritization and then introduce three different ways in which these problems can be handled.

The Ideal Prioritization Scenario

In an ideal world, we would have a magic number that assigns weight to every single story. Whenever we're picking up work to do, we simply rank by this number among all available stories, and pick the one with the highest value.

That would be simple, would it not? Everyone in your team agrees that higher is better, and the matter of why it is higher in value is well understood and doesn't need discussion.

Money is already an example of such a magic number. Whether it's Dollars, Pounds, or Rupees, the value of all goods and services are reduced to this number. While there are still markets where the price is negotiated for each transaction, markets have converged towards fixed, non-negotiable pricing as the most efficient way to handle routine day-to-day transactions.

In an ideal Prioritization scenario, every story would simply have a 💲 sign on it that denotes the value that can be generated from it, so that the value placed on each story by your team is aligned with the value placed on it by your market.

The Reality or Why Prioritization is Hard

Pricing is usually arrived at with a reference number - what your cost of production was, how your competitors are priced and so on. This reference number is then manipulated with a simple calculation, and we have our pricing.

Unfortunately, product development isn't always that easy. Here are the three primary issues that make the prioritization hard in order of importance:

Which of these three issues does your team run into most often? We've laid them out here, so that they become easy to identify and handle.

Apples and Oranges

This is the easiest of the barriers we can face, but still non-trivial. An example would be to prioritize an expected 2% increase in conversion rate against an expected 2% decrease in churn, or a 5-point increase in NPS against a 6% increase in Daily Engagement. Overall, impact for both problem and solution has been measured with high certainty, but we're comparing apples with oranges.

In larger organizations, this problem is solved through team structure. The kind of metrics any team deals with aren't this divergent, and so these trade-offs can be eliminated or made less drastic by design.

For teams where such clean division of responsibilities isn't possible or desirable, there is a potential solution for this problem (again) in the monetary system: exchange rates.

At large Tech companies with sophisticated Analytics operations, metrics around conversion, engagement and churn are deliberately made convertible to each other. This is a highly recommended solution that would greatly benefit organizations in the long-term.

Hard to Quantify

There are many cases where we may want to do something, and we would have strong conviction that it's an important story that would create a lot of value. But, we cannot quantify its expected impact.

Here's an example: let's imagine you're working on an app that uses a certain color scheme that is perceived as ugly. You know that this impacts the first impression of your app, and you also know that bright, friendly color schemes are all the rage right now.

How would you predict the impact of the change in color scheme? Would there be any change at all? Does it really matter? Is this perspective simply a distraction? Stories that fall in this category are often at a disadvantage over those whose expected impact can be quantified. This is also fertile ground for getting work done by pulling rank, or by "gut feel".

Uncertainty or Lack of Data

How certain are we about our expected impact? Let's imagine this scenario: Our app has a load time of 2 seconds between screens. Customer surveys of churned users have revealed this to have a 20% impact on churn. We propose to reduce load time to 1 second. But are we sure that would that be enough?

What if users want it to be under 500ms? Or the elusive 100ms? What if reduction to 1 second doesn't move the needle at all, and we could've done something else instead? Where should we draw the line? And how should we judge?

In this example, unlike the previous one, the impact of the problem is known. But the impact of the solution (still) is not.

How we can deal with it

Multiple frameworks have been devised to deal with these problems and usually solve one or more of these problems really well. Here, we take you through each of them.

Use Reinertsen's Formula

Donald Reinertsen's book The Principles of Product Development Flow analyzes Product Development in very good detail and was the basis of many of the logical and philosophical underpinnings to the Scaled Agile Framework.

This analysis is extremely sound, and offers a number of "principles" that can be utilized for managing Product Development teams. Here, we introduce the Weighted Shortest Job First formula as a method of prioritization. Before we dive into the formula, we introduce the two terms used:

➡️ Cost of Delay is the core of this way of thinking. It is so important, that the book says that if we measure one thing only, we should measure the cost of delay. But what is it, and how should we calculate it? Here's how we at Nirvana think about it:

If the feature improves conversion, ask: If we were to do it one month late, how many users would we not acquire in that month? It helps to know your LTV to assign the cost of delay.

If the feature improves engagement, ask: If we were to do it one month late, how many hours of engagement do we lose? It helps to know the monetary value of user engagement to arrive at the cost of delay.

If the feature reduces churn, ask: If we were to do it one month late, how many more users would leave, never to return? It helps to know your CAC & LTVs to get to the cost of delay.

With this, the features that have clear monetary impact assigned to them can be in their own tier, and those without would naturally be deferred. Why should this be the case? Here's a quote from the book while it attempts to answer this question:

"Most corporations give control over financial resources to people who worry about the economics of their choices. To influence these people, we must speak the language of economics, not the language of proxy variables.

When we speak to those who control the money, using the language of money, we can get fast decisions and enthusiastic support."

➡️ Effort Estimates are the second factor. We may know our estimates with a good level of certainty, and sometimes we may not. The principles account for both eventualities and recommend the course of action for all scenarios.

So, what's the formula?

It's quite simple.

Priority Score = Cost of Delay / Effort Estimate

In effect, this ensures that shorter stories with the same costs of delay get picked up first, while dealing with the heterogeneity of the Impact and Estimate that most stories would have.

There are also recommendations around how teams should deal with tasks whose estimate is unclear. In all, this book is recommended reading for anyone who builds products and is concerned with streamlining work inside their team or organization.

Do RICE Scoring

Intercom's RICE score is a simple way of dealing with these problems. It has four parts

➡️ Reach implies the number of people a given feature will affect in a given period of time. This, is an easy-to-compute, concrete number, which wouldn't be much of a bone of contention.

➡️ Impact, however, is notional. It does not directly deal with the problems listed above, especially with quantification and comparison. But it supports a multiplier after agreement is reached based on the below scheme:

0.25x (minimal) , 0.5x (low), 1x (medium), 2x (high) and 3x (massive)

➡️ Confidence is the same as explained before. It is expressed in percentages with the following suggested scale: 100% is “high confidence”, 80% is “medium”, 50% is “low”. Anything below that is “total moonshot”.

➡️ Effort is measured in "person-months". If a 4-member team were to work for 1 week each to build the feature in question, the effort score would be 1. This can get quite complicated on its own, which is why we wrote this guide to effort estimation.

The RICE Score, then, can be computed with the following formula:

RICE Score = Reach x Impact x Confidence / Effort Estimate

The quantification of "Impact" remains a bone of contention in this model, but RICE scoring remains a simple and easy way to reduce fog in the prioritization process.

Make it a Game

The "Buy a Feature" game is an interesting prioritization technique that's designed to be both fun and insightful. And, apart from being a prioritization technique, it is a great customer research technique as well.

This technique is explained in The Principles of Product Development Flow and while the formulation described below appears in Luke Hohmann's book Innovation Games.

Here's how it works:

➡️ Make a list of features

➡️ Set a price on each feature, based on the development costs of that feature

➡️ Invite a set of customers to a meeting and explain or sell each feature to them

➡️ Give play money to customers with which they can purchase any feature they like

➡️ Customers should be allowed to negotiate prices or to team up in order to buy features

➡️ Watch where the money goes and why

When participants have spent all their money, a discussion around all the purchase decisions ensues. This helps the Product Development team get a sense of how their users perceive the value of each of their features, and this also works as a rehearsal for the market response to each feature.

In Innovation Games, it is recommended that one or more of the features in the list should be priced high enough such that no one participant has the money to buy that feature on their own. This creates an opportunity for participants to speak with one another to make their purchase. The nature of such alliances would be very revealing of the value of the product.

The outcome of the Buy-a-Feature game is twofold:

First, teams can leverage their knowledge of the buyer's perception of value to arrive at the right fit and finish for their chosen features.

Second, they can use the "profits" directly for prioritization or as a strong input to their prioritization process

That is a short description of the "Buy a Feature" game. It's "Straight from the Horse's mouth" and it also "Shows you the money". Also, it can be a lot of fun.

Conclusion

What would be the best course of action for you? Here's what we suggest:

RICE Scoring is a great place to get started. It is the least difficult way to bring structure to your team's prioritization process.

Conduct Buy a Feature games with your users and also among team members whenever possible. It is a great input to the prioritization process and would go well with RICE scoring or with Reinertsen's Formula too.

As soon as it becomes easy to measure the economic impact of our stories, Reinertsen's formula becomes the easiest and most precise way to prioritize work.

Also published at here.