In this post, we study the Survivorship bias — the danger to concentrate your data analysis solely on existing power users — explain the potentially harmful consequences and present actions to overcome it. This article is part of a series about defeating bias as a Product Manager.

When focusing exclusively on current users' data, you harm your prioritization process.

Product Management is complex. You have to decide on the product’s fate while listening to various stakeholders with each different set of requirements. The prioritization of your roadmap is made even more difficult by conflicting sources of information. You might want to purely focus on your current user data usage to make “objective” decisions, but sometimes cognitive bias prevents you from thinking clearly.

Definition of Cognitive Biases:

“By cognitive bias, we mean cases in which human cognition reliably produces representations that are systematically distorted compared to some aspect of objective reality” (Haselton et al., 2015) [1]

Cognitive Bias — Why what your brain thinks is not true — Photo by TheImmeasurable

Being aware of how your brain can trick you will help you make a better decision. In this post, we look into the survivorship bias:

1) Survivorship bias intro

2) Survivorship bias in the Product Management field

3) Two real-life examples of Survivorship bias

1) The Survivorship Bias

1.1) World War II (WW2) bombers analysis

To introduce survivorship bias, we will present its most prominent examples. During WW2, engineers examined the planes that were dropping bombs coming back from the battlefield. Analyzing the bombers, engineers realized the aircraft had bullet holes everywhere except the cockpit and the engines.

Bullet holes pattern from the WW2 bombers analysis — Photo by SteveMurch

Based on this pattern, they decided to reinforce the aircrafts in the areas where they had the most bullet holes to make them bulletproof. From this analysis, they strengthened everything but the cockpit and the turbine. The problem was actually when a bomber was shot in those two sections it never made it back to its base as it crashed on the battlefield. Therefore, the analysis focused on survivors without considering the bombers buried in the sea. What they had to do is armouring the planes where the bullet holes were not.

WWII is a dramatic example of survivorship biases, but you will find more frequently than not real-life examples in your day-to-day as a product manager. It can often happen in your workplace and can have terrible consequences on your ability to make good decisions.

1.2) Survivorship bias definition:

The concentration on the people who “survived” a process and inadvertently overlooking those who did not survive because of their lack of visibility. (appliedframeworks.com )

For Product Managers with this bias, this implies an over-confidence in prevailing clients’ demands and preferences rather than examining the total addressable market requirements.

2) Survivorship bias in the Product Management field

Survivorship bias happens in Product Management when you concentrate your user research, testing, and analytics on users who are the most active (power users). The power users tend to be more positive about your products, they are easier to reach, and they are more visible. In the end, power users can over-represent their data.

Survivorship bias is a shortcoming for both Consumer and Enterprise companies:

Ken Sandy, a 20-year veteran in the consumer internet industry previously VP of Product at Lynda.com, surveyed consumer app and consumer application companies. He shared his findings during the INDUSTRY The Product Conference 2019 conference.

2.1) Survivorship bias within Consumer apps:

Behaviour analytics of current users is the number one source to prioritize product features in consumer companies. There is a wealth of data that allows you to understand how your active user interaction with your products deeply. But be careful it’s not sufficient.

• 57% of PMs agreed that they rarely or never participated in an interview with a dissatisfied customer
• 1/3 ranked disengaged users or elapsed customers data as a top 3 way to prioritize product features.

2.2) Survivorship bias within Enterprise app:

In the enterprise product, the findings are very similar: PMs are only looking at power users. The features prioritization is driven by existing users and the one that is more vocals and highly motivated.

• ⅔ ranked users' behaviour (user onboarding and users adopting and using the enterprise product) as within the bottom 3-way to prioritize the product feature.
• Only 12% of enterprise software PMs use disengaged customers’ data, churned customers or people who said no to your Product to prioritize.

An organization that purely look at active and happy customers miss out on an infinity of opportunities:

• An opportunity to understand how we can improve our Product
• A possibility to expand the use of our Product to those who currently don’t see value in it

PMs must take into consideration disengaged customers when Prioritizing their Backlog — Photo by visual-paradigm.

An organization that purely looks at active and happy customers misses out on an infinity of possibilities.

3) Two real-life examples of how to overcome Survivorship bias

3.1) The eBay case study: searching for a dress

“The dress” — when words are not enough to describe and search for a specific dress — Photo from Wired

Clémence Tiradon, Director of Product Management at eBay, described an obvious example of overcoming this bias.

During a conference, she asks the audience to raise their hands if they ever bought clothing on eBay. The problem, she explains, is that PMs tend to focus on the people who raise their hands and are proud of themselves for having so many people within the audience who are eBay clients. Instead, what PMs should do is focus on people who didn’t raise their hands and ask them why. There could be a million reasons for not using eBay: not relating buying clothing with eBay, or trying to buy apparel on eBay but failing in the process. But you never know the real reasons until you start having a difficult conversation with your non-customers.

Your target population is not people wanting to buy a dress on eBay; your target population is anyone who wants to buy a dress online.

Prioritizing features based exclusively on current users' analysis might prevent you to capture new users — Photo by Go-oodles.

Clémence Tiradon, describes 3 ways to overcome the survivorship biais.

A) Using multiple data source:

You need to try to get another perspective by looking around. That means to use various data sources, this is critical. You cannot rely only on your quantitative data — your metrics from your Mixpanel or Firebase dashboard. You cannot either solely focus on quantitative data from your user testing and research. You don’t want to make a decision entirely based on a significantly reduced set of customers. As PMs, you make the best decisions when you overlap quantitative and qualitative insights to understand your customers’ main drivers and understand their experience and how they are experiencing your Product.

B) Understanding the context of where the data is taken from:

A common misconception is that data is an unbiased fact, instantly available to use. Data is not neutral; it has been collected and inherits views from people who intentionally or not collected it. Without context, data is worthless as well as any conclusion you drive from this data.

In real life, statisticians don’t question “What is in the numbers?” but “What does the data represent in the world; and how does this relate to other data?”. (BigThink).

You need to understand the context: when was it? What is the mood of the interviewees? Understanding how the data has been gathered, and the customer state in mind when it was gathering will help expand your field of vision.

C) Ask “What If” scenarios

“What if” scenarios are critical to start to expand that field of vision. If we come back to the dress example, what if the customer looking for a dress is not an English native speaker and can’t find the exact word to describe the dress (or can’t agree if it’s a blue or gold dress)? What if they find the dress looking at someone passing from the street? What if they can take a picture? What if instead of describing the dress by a word, they could use that image to search?

By expanding your field of vision, you allow yourself to bring more insights into the decision making for your Product, and you will come with more innovative solutions that fit closely with your users’ needs.

3.2) Example of a Sales-driven organization

Survivorship bias is particularly prevalent in Enterprise Software and Sales Driven organizations.

Those organizations have common characteristics :

• Sales-driven culture: they focus on what would be the next deal.
• Customer responsive: they came in; they understood the different customer needs; they make sure they addressed those needs regularly with the product releases.
• Urgent fixing the customer problems always outweigh the long term strategy of where the company needed to go.
• Trying to serve everyone and successfully serving no one.

Changing how you deal with clients’ requests to overcome the survivorship bias:

• Focus on a few segments only: focus on one prominent segment that is attractive. The sales team might have to encourage out smaller, more demanding customers. This prevents you from trying to please every single existing customer (profitable or not).
• Structure how you bring work into the organization: from merely looking at customer requests to dividing into various streams of works one being customer requests (only a small part of the organization would work on that). Separating customer requests from other streams allows to create a more robust prioritization against resource capping for these customer requests and free up a lot the organization to start thinking strategically on how to serve more extensive segments.
• Create a Roadmap according to your long-term Product vision: The new prioritization process allows creating a Product Roadmap that is closer to your company's long-term vision instead of a clients’ Wishlists.

Summary — Actions to avoid survivorship bias:

1. Use multiple data sources
2. Understand the context of the data
3. Create alternative scenarios
4. Focus on vision and where you want to go instead of who your current users are
5. Assign clients request within a particular section of the backlog to track time dedicated to this section
6. Create a Roadmap that reflects your Product vision instead of clients’ wishlists.

Congratulations if you made it this far! I hope this article helped you in understanding better what a cognitive bias is and how to overcome it.
I would love to hear about your real-life experience with Survivorship bias.

And don’t forget to check my other posts about Product Management. I wrote a series about cognitive bias.

Lead Photo by MaisonDigital