Survivorship Bias in Startups

An amazing story from World War 2 provides a lesson in user acquisition for startups.

During World War 2, the allied military wanted to add some more protective armour to their planes.

However, they could only add a limited amount of armour to each plane before they became too heavy to fly.

This meant that they had to choose specific parts of the plane to attach armour (and leave un-armoured) to optimise the number of planes returning home.

Collecting data

They began by gathering data and analysing the locations of the bullet holes on all the planes in their fleet. In this way, they could see where the best place would be to attach the extra armoured plating.

Their initial idea was to attach armour to the places which had been, on average across the whole fleet, the most peppered by bullets. Seems logical.

They found that there were more bullets holes per square foot in the wings and the fuselage than in the engine and fuel system. So the initial thought was to add armour to the wings and fuselage.

Not what Abraham Wald thought…

However, a mathematician named Abraham Wald, quoted by many as being the cleverest man in any room, saw the situation differently.

He made a brilliant yet incredibly simple observation, which resulted in them attaching armour in the complete opposite way to what you would expect.

He advocated that the military should attach armour to all the places where there were few or no bullet holes across all the planes in their fleet.

Why?

Well, all of the planes that they were analysing had come home safely. The locations of the bullet holes in these planes, therefore, were not fatal blows since they hadn’t brought these aircrafts down. These planes had managed to get home.

This meant that the areas in these ‘surviving’ planes where they found no holes must be the places where the planes that weren’t so fortunate took fire.

“The missing bullet holes were on the missing planes”-Jordan Ellenberg

This brilliantly simple observation completely reversed the approach the military took towards armouring their planes, and no doubt saved countless lives.

This story is an example of survivorship bias, where the data set analysed is only taken from the success stories.

When I read this story, it immediately made me rethink our user acquisition approach at Pilcro.

Survivorship bias in startups

The following example relates to my experience at Pilcro, a software startup, but it can be applied more generally.

Imagine each potential user in your target market is a plane.

Then every sticking point, bad advert, miscommunication, missed conversion, bad meeting, missed opportunity in your marketing and sales funnel, confusing moment in your user onboarding, is an enemy bullet.

And this makes your top users those planes that made it home safely with only non-fatal bullet holes received.

Where does your feedback come from?

Now the problem is that these top users, those who you have already sold to, those who have flown all the way down your marketing or sales funnel and now use your product regularly, are the ones from whom it is most easy to gather feedback.

But any product related issues that these users have at this point are simply bullet holes through the wings. They have bought your startup’s product and their plane has landed.

Now, in an ideal world, your top users wouldn’t have a single complaint about your product, and, of course, if they do, it shouldn’t be discounted. Customer success is extremely important in the early days of a startup.

But, in these early days, having a faultless product and a faultless marketing and sales funnel is wishful thinking.

So the question is… where should you attach your armour right now? Where do you focus your startup’s limited resources?

Feature Creep

A consequence of survivorship bias in many software startups is feature creep.

Feature creep starts with feedback collected off those from whom it is easiest to collect it: your current top users.

By asking current active users what improvements they would like to see in your product, you think that, by building new features, many more planes will soon be flying safely down the marketing and sales funnels and into the product.

But this might be entirely missing the reason why those in your target market who haven’t given you any data at all are not interested in your product in the first place.

There may be an enemy machine gun which you haven’t even identified, bringing down a whole range of planes from your target market, depriving you of their data and feedback.

Journey of a potential user from nothing to top user

We had exactly this realisation

At Pilcro, we were asking all of our current users what they would like to change in our app because they were the easiest people to ask. But we weren’t addressing why large parts of our target market weren’t giving us any feedback at all.

We weren’t going out and looking at the planes that had been shot down, the gaps in the data.

We came to realise that we had acquired a large proportion of our users by speaking at events, talking and sharing our product in person. Direct sales.

However, our marketing website was not giving us as much data as we wanted and we weren’t making enough conversions from it. We were so focused on the success cases we already had that we weren’t thinking about why our website was depriving us of more users.

Countering survivorship bias

To counter this survivorship bias, we took the following steps.

1. We worked out which parts of our target market were giving us no feedback. We did this by looking at where our current users sat in our target market and how we had acquired them.
2. We got in touch and gathered feedback from some of those who had seen and ignored our marketing and sales outreach online.
3. We got in touch and gathered feedback from some of those who had visited our website but had not begun a free trial.
4. We have now put in place measures to gather at least some feedback from the people who are not interested in our product at each step of the above diagram. Negative feedback at any stage of your funnel is the best way to learn!

What we changed

We discovered that people were finding our marketing website a bit confusing. It needed to be updated.

So we chose, based on this recovered data, to redesign our website to be much more interactive. This meant visitors could learn more about Pilcro while they were still deep in enemy territory.

Specifically, we built a Pilcro Artboard auto-generator, to showcase what our product had to offer as quickly and clearly as possible.

And, by doing so, we have been able to find out so much more about who is visiting our website and why. What’s more, we now have 3 times more conversions from website hits to users (based on week by week data from Google Analytics from before and after making the changes).

3 times more planes landing safely.

Next step is to redesign and refine the marketing material based on the feedback from those who weren’t interested!

If you enjoyed this blog, please clap and follow me on Twitter. Thanks to Dr David Slavin for introducing me to the story.