Validating an Idea is Hard: Part 2

When you’re excited about an idea, feedback from strangers can be really misleading.

A few weeks ago I wrote about my early failures validating my placecards idea. When I sat down to write, my plan was mostly to just tell the story of getting kicked off forums and threatened by websites for self-promotion, but as I was re-reading the conversations something interesting happened:

I interpreted the feedback I got completely differently.

It had only been a month and yet I felt like I was reading things as an entirely different person! How and why that happened — and what can be learned from it — is the subject of this post.

Problem 1: You Can’t Trust Anyone

So one big thing that happened in that month is that I read a fun and helpful little book called the Mom Test, by Rob Fitzpatrick. The subject of the book is “how to talk to customers and learn if your business is a good idea when everyone is lying to you.”

The key part is that everyone is lying to you.

Most people don’t necessarily know they’re lying to you — a few might — but people do lie, mostly unintentionally. Common reasons for this include:

1. They don’t want to hurt your feelings
2. They are rooting for you to succeed
3. They are bad at predicting human behavior

The most common way that people lie is when it comes to money. Let’s look at a few common fallacies/lies related to this topic.

The “I would pay for this” fallacy

People love saying they would pay for things, especially things in the ideation phase.

Why is that?

Because it’s free!

Saying “I would pay for this” costs you nothing. $0. Virtually 0 time. And in the process you get to see someone get excited and encourage them to follow their dreams.

Ironically, the reason it’s so easy to say you would pay for something is that you are not actually paying for it.

So, you can’t actually interpret “I would pay for this” as willingness to pay. How do you test it, then?

You have to actually try to take their money.

Here’s how that has typically played out for me:

Possible Customer: I would totally pay for this.

Me: Great — it’s $10, we can take care of everything right now!

Possible Customer: Hmmm… well I’m not sure I’m ready to do that just yet….

Me: Why not? You just said you’d pay for it.

Possible Customer: Well… I’d need to know whether [it does something that it doesn’t do yet] and I’d want to see what other options are out there…

Let’s take a look back at some of those those forum posts that I thought validated my idea. Armed with my newfound perspective I can see that they are littered with the “I would pay for this” fallacy.

So here we see someone saying they would pay for my service. Sweet! Right?

Well, no, not really.

If we look closer, we can see multiple reasons that this person is lying:

Reason 1: She does calligraphy so would never use this service.

Reason 2: She’s already married! Definitely not buying place cards any time soon.

Conclusion: don’t trust when anyone says “I would pay for this”, only trust people giving you money.

The “others would pay for this” fallacy

A close cousin of the “I would pay for this” fallacy is the “other people would pay for this” fallacy. In fact, in my case this was almost always what people actually meant when they said they would pay for my product. Here’s a perfect example of this in the wild:

“Others would pay” is even worse than “I would pay” if you don’t train yourself to recognize it. It can feel a lot like validation but in fact it should be even less confidence-inducing than “I would pay”. This is especially true because it’s harder to refute — people are just stating an opinion that’s just as uninformed as your own.

When someone says “other people would pay for this” what they are really saying is “I hope you succeed and in the three seconds I spent thinking about it I don’t think that’s impossible.”

Doesn’t really feel like validation when it’s put like that, does it?

Problem 2: You Can’t Trust Yourself

So now you know that you can’t trust anyone, but that’s actually only half the problem.

The other half is that you can’t trust yourself.

You are asking these questions of people because you want to get a certain answer and as humans we are very good at filtering out the information that conflicts with our own beliefs and embracing the information that confirms it.

You might recognize this phenomenon as confirmation bias, and it’s so common that there’s an entire field of study devoted to it.

Confirmation biases contribute to overconfidence in personal beliefs and can maintain or strengthen beliefs in the face of contrary evidence.

— Confirmation Bias, Wikipedia

When you are trying to validate your idea you’re extremely prone to the perils of confirmation bias.

Seeing it in action

So going back to my own idea, when I first started posting on forums and sharing the idea with other people I had one and only one goal: to confirm that it was a good idea.

This made it really easy to quickly skip through and dismiss any comment that didn’t fit my own narrative of the idea, and embrace any comment (typically one of the fallacies mentioned above) that confirmed I was onto something.

Armed with a bit of distance and a bit more knowledge I can revisit some of those early forum postings and see how my perspective changed from my early, naive, optimistic days to the slightly-less-naive but still-probably-totally-ignorant perspective I have now.

Here’s a post I completely glossed over the first time around from a thread on the wedding bee forums.

By saying that printing place cards at home by hand without my tool is “very easy”, this person is basically saying that the problem I’m trying to solve doesn’t exist.

There’s tons of comments like this sprinkled throughout the threads. For example here are some more quotes from the thread on wedding wire:

• “It took an hour to print them, tops”
• “We made it. It wasn’t hard.”
• “If you buy place cards from Minted they print them for free and shipping is pretty fast.”

All of these people were basically saying “this isn’t a real problem for me”, and they are worth listening to.

Conclusion: Take the time to read and process every piece of feedback, positive and negative, and be very aware of your own confirmation biases.

So how do you actually validate ideas?

When I showed a draft of this piece to my wife she responded with the following:

My biggest question after finishing this article is: “Ok, it’s all hard. So… what do I do? Is there even a point to talking to users then?” Curious what your actual takeaway is.

What is my actual takeaway?

I stewed over that for a while, thinking that “I am clueless” didn’t sound like a great takeaway, and ended up with the following sentence:

It is far better to validate ideas with information than with opinions.

I did get lots of valuable information from the forum posts, but it was through the information that was conveyed, not the opinions people had. For example, one thing I learned from the forums was that loads of people — probably about a third of people who responded — were printing place cards at home. This meant that providing a print-at-home based service was not going to fail because there was no demand for printing at home.

I also learned things about the market, like that sometimes place cards are handled entirely by wedding planners. Again, this is valuable information and opens up a new audience for me to try and target.

On the pricing front you can go from opinion to information in a very simple way — don’t ask them if they would pay, ask them to pay. That’s the only way to know if they actually will.

So yes, people will lie to you if you open up a door for them to lie, but only if you ask them for their opinions.

If you ask them about facts — or find a way to convert their opinions to behavior — you are much more likely to learn the truth.

Originally published at www.coryzue.com.

If you liked this post I’d highly recommend reading the Mom Test, which goes into this stuff in a lot more detail. I’d also appreciate if you would click the green heart to help others find this. Thanks much.