Product Experimentation Wonderland

In “Alice in Wonderland” we are invited to escape reality by tumbling into a whimsical world of nonsense. I found in this story a good analogy for what happens in the life of product managers.

As PMs, experimentation is one of the most important activities we do. As we explore several alternatives at the same time, we collect more and more insights about the underlying problem we want to tackle.

However, sometimes it’s easy to fall into the trap of extending our “experimentation” for months or even years! Instead of multiple “cheap” experiments, we decide to invest in MVPs or POCs that live much longer than they should.

The effort to build a basic experiment turns out to be not as simple as we thought, and your favorite MVP or POC becomes more like a liability than a valuable asset.

"'Who in the world am I?' Oh, that's the great puzzle!"

In this article, I will provide you with some basic checklists I strongly recommend you use every time you start and go through an amazing experimentation adventure.

Alice’s wonderland

There are marvelous and exciting stories of successful experiences and awesome products that made companies successful. At the other end of the spectrum, when Alice is lost in a mockery world, you find the cases that have failed catastrophically or even ridiculously.

In the middle is where most of us will live and work our entire lives as product managers. Trying, failing, succeeding sometimes, and ensuring that what we have learned from the failures of the past can be somewhat valuable for future successes.

If you don’t work in a true product company, the odds are that this learning process may be long and difficult. Sometimes it is only up to you to make sure it’s safe to explore and dream.

Luckily, the vast majority of companies today have an agile mindset. However, in practice, how many times are they escaping reality and living in a world of imagination?

"It takes all the running you can do, to keep in the same place. If you want to get somewhere else, you must run at least twice as fast as that!"

Down the rabbit hole

No project, idea, MVP, or “experiment” gets to a disruptive conclusion (the sudden awakening) without having passed by several moments of decision making. Those moments either reinforce the idea to continue investing and keep the dream alive, or they were simply ignored or overlooked by Product Managers (and the entire team in general).

“Alice follows the rabbit because she is 'burning with curiosity.' Soon she finds things becoming 'curiouser and curiouser.'“

I truly believe that this doesn’t happen intentionally. On the contrary, during the entire period, the team keeps engaged, excited, and eager to see the expected results. However, it is not sustainable, productive, or in line with the best product development frameworks.

There might be some reasons to explain this day-dreaming, counter-agile mindset:

• You are fed by one or two stakeholders that are really excited about your experiment.
• You and your team fall in love with what you have built and desperately try to make it work.
• You get too close to see it from a cold and distant perspective — You lose your scientific approach.
• You have fallen into the build trap.

At some point, someone decides to stop the madness and end the “experiment”. Hopefully, you are aware and can make that decision by yourself. But that’s not always the case...

So, how can we avoid the rabbit hole or even be able to know when we have started living in a magical wonderland?

Will you fall asleep?

It all starts when you are lying down under the tree of problems and looking up into the opportunities and new ideas…

“Why, sometimes I've believed as many as six impossible things before breakfast.” (Alice)

This moment is very well studied in the field of Product Management, and I see this across different companies from all the business areas. We tend to predict amazing results from little investment and start believing that we have found the perfect solution for the problem. We just need that. A “little investment”...

To be more specific, I suggest you have a checklist every time you start a new experiment or discovery period:

• [ ]undefinedYou know exactly what hypothesis you want to validate with your experiment.
  • [ ]It’s measurable and has target SMART goals.
• [ ]undefinedYou have defined end date for the experiment (it’s the “T” in SMART, but make sure you don't forget that resources are limited — one idea may be the “metered funding” approach).
• [ ]undefinedYou already have in your mind the next experiment you want to do if this one fails.
• [ ]undefinedYou are not biased towards the experiment outcome (you can have expectations as long as you are aware of them and try your best to act like a true scientist and not influence the experimentation environment).
• [ ]undefinedYou have defined your target audience and are aware of the survivorship bias (example: consider customers/users that do not use/know about your product. Include the perspective from the ones that have churned. You have failed to acquire, or even using direct competition or alternative solutions).

If you check everything, you are good to go. You can start exploring, but be cautious because you might go down the rabbit hole at any moment...

Reality check

Once you start an experiment, you’re eager to find the results (and sometimes to confirm your previous assumptions). That’s the perfect time to ask yourself: “Am I in the real world, or am I dreaming?” In other words: should I continue investing in the experiment or not? Am I (unconsciously) forcing the experiment to get the results I strongly believe?

The answer really depends on the hypothesis you have defined at the beginning of the exploration phase. Pause for a second and evaluate if:

• [ ]undefinedYou have got all the answers you need, meaning:
  • [ ]You have all the results from your previously-defined SMART goals
  • [ ]You have confirmed the four big product risks (have a clear yes or no response to value, usability, feasibility, and business viability)
• [ ]You have avoided survivorship bias
• [ ]You are not emotionally attached to the results, no matter what they were

If that’s the case, great! You are ready to make the right, conscious, and supported decision, be that to keep investing or to try another path.

“Really, now you ask me,” said Alice, very much confused, “I don’t think—” “Then you shouldn’t talk,” said the Hatter

But, once again, beware of the rabbit hole… Especially when the experiment gets you different results from the ones you were expecting. That’s why the last check from the list is probably the trickiest one.

For better or worse, it’s up to us, Product Managers, to keep it real. See failures when they happen and do not persist in them.

Furthermore, these cases are golden opportunities to learn: inspect and get insights. Conduct a group post mortem and — as a team — look back and document why the experiment didn't achieve the expected results. (Were each team member's expectations different? Were we too ambitious? Did we forget or ignore a basic premise? How can we make sure this doesn't happen in the next experiment?)

Wonderland is a world of discovery

As PMs, it’s so easy to fall in love with the problem you want to solve. So there is an enormous responsibility on our shoulders. It requires us to sometimes be cold-hearted to make decisions — stop dreaming and go back to reality.

In the end, Alice truly wakes up from her dream. She realizes how amazing it was and keeps dreaming of what the future will look like.

`Wake up, Alice dear!' said her sister; `Why, what a long sleep you've had!' `Oh, I've had such a curious dream!' said Alice

“Alice in Wonderland” is a story of growth and progression through life. For PMs, experimentation can still be a true wonderland, but above all, it should be a great learning experience and awareness practice.