Product Hypothesis Validation: Best Practices & Examples

As there are more companies emerging each day, product managers are trying to look for innovative features to implement in order to stand out among competitors and help boost user engagement. But how do you know which features your product users want to get? This is where product hypotheses come in handy. These experimental solutions offer a great approach to ensuring that your product aligns with customer needs. However, many businesses suffer great losses simply because they do not know how to validate hypotheses in the right way. This article discusses a framework for generating your own product hypotheses so that your business is based on facts – and not on assumptions. But first, before we get started, let’s break down what a product hypothesis actually means. A product hypothesis is a statement expressing an assumption, used as a tool to test and validate ideas about your customers’ wants, needs, and/or values, and how your product can deliver all of them. In general, hypotheses are used by product managers to make or discard market decisions and prioritize activities based on their impact on the product. Imagine that your idea is to create a platform for buying and selling used objects – something similar to eBay. Your hypothesis could look like this: if we allow free shipping for a small category of products (Variable A), it will drive up user activity in this category by 10% (Variable B). The hypothesis contemplates two variables that are related, where if variable B results in an increase of more than 10% we can define it as a valid hypothesis – otherwise, it has been refuted. Nevertheless, it is common to confuse the meaning of product hypothesis with the concept of an idea, with the latter having no need to be verified and being a starting point with a general vision for the product or service. Innovation experts such as Michael Schrage also draw a distinctive line between the two terms. Schrage, who has served as an advisor on innovation issues and investments to major firms, including Mars, Procter & Gamble, Google, Intel, BT, Siemens, NASDAQ, IBM, and Alco, defines the concept of hypothesis as something that does not only need to be tested but also has, by its nature, an objective that can be tested; whereas an idea is a completely subjective variable. Sounds difficult? Take a look at these examples: Examples of an idea: Why don't we implement sales notifications via WhatsApp to the platform for buying and selling objects? Let's organize an interview with the CEO of our company and broadcast it on the most-viewed TV channel in our town. Examples of a hypothesis: If we implement sales notifications via WhatsApp to the platform, we will see an increase in the use of our API linked to the WhatsApp platform. If we organize an interview with our CEO on ITV, 5200 people will engage with the interview, and we will increase our marketing traffic by 10%. Additionally, I want to stress that not all product hypotheses require testing, but they should always be inherently testable. Step 1. Define the Objective One of the main foundations for the development of a product hypothesis is the problem statement, which involves a brief and clear description of the objective to be addressed. This objective description – specific, flexible, and data-driven – serves as a guide for the design of your hypothesis experiments. For instance, if you run a platform for buying and selling used goods, a low user retention rate (URR) indicates that your users are not satisfied with the product. How can you improve your metrics without losing the current clients? The URR metric should be analyzed as a whole, taking into account other important metrics such as percentage of active users (PUA), Conversion Rate, Customer Acquisition Cost, Customer Feedback, etc. A clear vision of the problem allows you to tackle the problem in a strategic way. Some possible strategies for solving this problem could be improving user experiences, providing quality content, encouraging user engagement, and monitoring and analyzing data. Step 2. Identify Variables The variables of a hypothesis can be classified as dependent and independent, both of which are quantitative, i.e., they are measurable. You can describe the independent variable as a cause that you can alter in your own way, and the dependent variable as the effect, which you will observe in the result. Independent variables are any changes in your product, whether it is a modification of the search engine, home page, payment interface, etc. This kind of variable modification will have an impact on the customer, and you will be able to observe it through dependent variables, such as the number of registrations, number of transactions, number of purchases and sales, number of active users, and so on. When a hypothesis is formulated with poorly defined variables, it is not clear whether it creates a problem or a solution. Let's take a look at this example: "Users do not use the platform because it is difficult for them to sell their products" – in this case, the variables are qualitative and very vague, there is no way to measure them. Instead, the correct way to form a hypothesis would be: "If we offer a free delivery service for a small category of products, the sales rate will increase by 15% over the period." Step 3. Make the Connection It does not matter if your variables are well-defined – if there is no logic and clarity between them they will not be able to give you any reliable results. 1 - Your variables could have a weak relationship between them. For example, a hypothesis that states "an increase in the number of products that the user wants to sell will lead to an increase in product sales" implies a low relationship between the variables as they depend massively on the cost and condition/likeability of the product. But you could create a much stronger relationship between the variables if you changed the hypothesis to "our top-rated sales category will offer free delivery for a period of time, which will increase sales by 10% in this category over said period” 2 - The relationship is plausible. One of your variables could be based on a vanity metric – meaning that the variable looks great at first glance but does not carry any meaningful results for your product. For instance, let's say the hypothesis is stated as follows "If I increase the number of followers on my social media, then all these followers will use my product that I advertise on my social media.” The independent metric has no direct correlation because the increase in social media followers is due to the content of your social networks – and not necessarily the likeability of your product. Step 4. Set the Validation Criteria Make sure that your hypothesis is built on realistic and measurable criteria. Let’s take a look at a couple of examples. First, you could use the ratio between active users and new users, where an index greater than 1 indicates an active use of the platform, and an index below 1 indicates that users are losing interest in the platform, as your confirmation criteria. Another criteria example could be the conversion rate (CR), which measures desired actions such as making a purchase, a ratio between those who see an object and those who wish to buy it. A high CR indicates that the product is desired or can be visualized as a desired category. A low CR may indicate that the products are not easy to sell. These examples show a strong correlation between their variables. Not all hypotheses are the same. They can be classified into null hypotheses or alternative hypotheses; the first being statements that negatively refute a hypothesis. For instance, “the number of active users will not increase if we implement a free delivery system", and the latter aims to prove the validity of a hypothesis in a positive way, "a free delivery system will allow the number of active users to grow." Each hypothesis statement is different and may vary according to its subject, tone, and wording. It should be noted that a hypothesis statement should always have two or more variables and a connecting factor. Once you are clear on these points, you can focus on assigning priorities to each hypothesis. This will allow you to focus on the most impactful solutions in order to create the product roadmap, reduce your to-do list, and choose a product hypothesis to test. Step 5. Testing Hypothesis Have you selected your final product hypothesis? Then let's put it to the test! It’s important for testing results to be as informative as possible. What’s more, the testing process is a continuous process that helps you learn more about your product and assumptions. Naturally, the most difficult part here is to decide which experiments to run. Remember, that you do not need to resort to software development, especially if you are running a start-up company and do not have resources to waste. Instead, you could use existing marketing tools to help you gather the required data. Landing Page Testing One of the popular ways to test hypotheses used by product managers around the world is creating a separate landing page for each experiment. Choose your target audience, and assign a distinct pain point to it. The more options you have, the better. But remember that there should be only one pain point assigned to each target - this will make debunking your testing results a lot easier. Create a landing page and a marketing campaign to fit each pain point. The landing page has to include a distinct design, pricing, and offer. You can change your variables depending on what you want to test. Place an opt-in form at the bottom of your landing page so that your target customers do not feel obligated to use your product or do not think that you want to obligate them. Instead, your goal is to collect honest feedback. You could also place a mini questionnaire inside your opt-in. Allocate the same budget to all the landing pages (hypotheses) you want to test. Compare the results using the same marketing funnel for all product hypotheses. A/B Testing Also known as split testing, A/B tests are a popular tool used by product managers and marketers to test all kinds of hypotheses. The point of the A/B test is to create two or more versions of the hypotheses, collect information about how users react to them, and choose the best option. For instance, if you want to implement free delivery for a specific category of products, you could test your hypothesis on whether it is better to choose the “beauty” or “clothing” category for the free shipping experiment via an A/B test. CustDev Interviews Last but definitely not least, simply ask your users what they want. CustDev interviews are a timeless way to strengthen your hypothesis test and a perfect step to start with. Yet, the instrument does not validate hypotheses quantitatively, if you find common patterns after 10+ interviews, you can further test it with quantitative methods, specify test details, or switch to another hypothesis without wasting time. CustDev is an in-depth interview with your clients, conducted online or in person in order to acquire as much feedback as possible. These kinds of interviews require a lot of effort to prepare but it is still an excellent way to quickly learn about your product assumptions. Conclusion Time to Take Action Now that you have finished testing your hypothesis, it is time to evaluate the data. But before you do that, I recommend answering the following questions: How well was the test executed? Is my data clean and up-to-date? Do I need more time to continue the experiment? If you are not sure whether your test was successful, here is something you can keep in mind in order to validate your product hypothesis in the right way: Take maximum precautions when gathering data. Create analytical dashboards, receive consent from your clients to record interviews, and write down notes. Do not waste too much time on one experiment. You are likely to conduct a couple of experiments to test one hypothesis, but if your experiments are constantly not working out, then maybe you’d better change your hypothesis to get clearer results. Do not waste your time for too long. Choose the right target audience. The right audience is the foundation of your test, therefore, you must be specific about who you want to test your product on and avoid biases and/or irrelevant testing “just to validate your hypothesis” at all costs. As there are more companies emerging each day, product managers are trying to look for innovative features to implement in order to stand out among competitors and help boost user engagement. But how do you know which features your product users want to get? This is where product hypotheses come in handy. These experimental solutions offer a great approach to ensuring that your product aligns with customer needs. However, many businesses suffer great losses simply because they do not know how to validate hypotheses in the right way. This article discusses a framework for generating your own product hypotheses so that your business is based on facts – and not on assumptions. But first, before we get started, let’s break down what a product hypothesis actually means. A product hypothesis is a statement expressing an assumption, used as a tool to test and validate ideas about your customers’ wants, needs, and/or values, and how your product can deliver all of them. In general, hypotheses are used by product managers to make or discard market decisions and prioritize activities based on their impact on the product. Imagine that your idea is to create a platform for buying and selling used objects – something similar to eBay. Your hypothesis could look like this: if we allow free shipping for a small category of products (Variable A), it will drive up user activity in this category by 10% (Variable B). T he hypothesis contemplates two variables that are related, where if variable B results in an increase of more than 10% we can define it as a valid hypothesis – otherwise, it has been refuted. if we allow free shipping for a small category of products (Variable A), it will drive up user activity in this category by 10% (Variable B). if we allow free shipping for a small category of products (Variable A), it will drive up user activity in this category by 10% (Variable B). T Nevertheless, it is common to confuse the meaning of product hypothesis with the concept of an idea, with the latter having no need to be verified and being a starting point with a general vision for the product or service. Innovation experts such as Michael Schrage also draw a distinctive line between the two terms . draw a distinctive line between the two terms draw a distinctive line between the two terms Schrage, who has served as an advisor on innovation issues and investments to major firms, including Mars, Procter & Gamble, Google, Intel, BT, Siemens, NASDAQ, IBM, and Alco, defines the concept of hypothesis as something that does not only need to be tested but also has, by its nature, an objective that can be tested ; whereas an idea is a completely subjective variable. Sounds difficult? Take a look at these examples: defines the concept of hypothesis defines the concept of hypothesis objective that can be tested Examples of an idea: Examples of an idea: Why don't we implement sales notifications via WhatsApp to the platform for buying and selling objects? Let's organize an interview with the CEO of our company and broadcast it on the most-viewed TV channel in our town. Why don't we implement sales notifications via WhatsApp to the platform for buying and selling objects? Why don't we implement sales notifications via WhatsApp to the platform for buying and selling objects? Let's organize an interview with the CEO of our company and broadcast it on the most-viewed TV channel in our town. Let's organize an interview with the CEO of our company and broadcast it on the most-viewed TV channel in our town. Examples of a hypothesis: Examples of a hypothesis: If we implement sales notifications via WhatsApp to the platform, we will see an increase in the use of our API linked to the WhatsApp platform. If we organize an interview with our CEO on ITV, 5200 people will engage with the interview, and we will increase our marketing traffic by 10%. If we implement sales notifications via WhatsApp to the platform, we will see an increase in the use of our API linked to the WhatsApp platform. If we implement sales notifications via WhatsApp to the platform, we will see an increase in the use of our API linked to the WhatsApp platform. If we implement sales notifications via WhatsApp to the platform, we will see an increase in the use of our API linked to the WhatsApp platform. If we organize an interview with our CEO on ITV, 5200 people will engage with the interview, and we will increase our marketing traffic by 10%. If we organize an interview with our CEO on ITV, 5200 people will engage with the interview, and we will increase our marketing traffic by 10%. If we organize an interview with our CEO on ITV, 5200 people will engage with the interview, and we will increase our marketing traffic by 10%. Additionally, I want to stress that not all product hypotheses require testing, but they should always be inherently testable. Step 1. Define the Objective Step 1. Define the Objective Step 1. One of the main foundations for the development of a product hypothesis is the problem statement, which involves a brief and clear description of the objective to be addressed. This objective description – specific, flexible, and data-driven – serves as a guide for the design of your hypothesis experiments. For instance, if you run a platform for buying and selling used goods, a low user retention rate (URR) indicates that your users are not satisfied with the product. How can you improve your metrics without losing the current clients? user retention rate (URR) The URR metric should be analyzed as a whole, taking into account other important metrics such as percentage of active users (PUA), Conversion Rate, Customer Acquisition Cost, Customer Feedback , etc. A clear vision of the problem allows you to tackle the problem in a strategic way. Some possible strategies for solving this problem could be improving user experiences, providing quality content, encouraging user engagement, and monitoring and analyzing data. percentage of active users (PUA), Conversion Rate, Customer Acquisition Cost, Customer Feedback Step 2. Identify Variables Step 2. Identify Variables Step 2. The variables of a hypothesis can be classified as dependent and independent, both of which are quantitative, i.e., they are measurable. You can describe the independent variable as a cause that you can alter in your own way, and the dependent variable as the effect, which you will observe in the result. Independent variables are any changes in your product, whether it is a modification of the search engine, home page, payment interface, etc. This kind of variable modification will have an impact on the customer, and you will be able to observe it through dependent variables, such as the number of registrations, number of transactions, number of purchases and sales, number of active users, and so on. When a hypothesis is formulated with poorly defined variables, it is not clear whether it creates a problem or a solution. Let's take a look at this example: "Users do not use the platform because it is difficult for them to sell their products" – in this case, the variables are qualitative and very vague, there is no way to measure them. Instead, the correct way to form a hypothesis would be: "If we offer a free delivery service for a small category of products, the sales rate will increase by 15% over the period." "Users do not use the platform because it is difficult for them to sell their products" "If we offer a free delivery service for a small category of products, the sales rate will increase by 15% over the period." Step 3. Make the Connection Step 3. Step 3. Make the Connection It does not matter if your variables are well-defined – if there is no logic and clarity between them they will not be able to give you any reliable results. 1 - Your variables could have a weak relationship between them. For example, a hypothesis that states "an increase in the number of products that the user wants to sell will lead to an increase in product sales" implies a low relationship between the variables as they depend massively on the cost and condition/likeability of the product. 1 - Your variables could have a weak relationship between them. But you could create a much stronger relationship between the variables if you changed the hypothesis to "our top-rated sales category will offer free delivery for a period of time, which will increase sales by 10% in this category over said period” 2 - The relationship is plausible. One of your variables could be based on a vanity metric – meaning that the variable looks great at first glance but does not carry any meaningful results for your product. For instance, let's say the hypothesis is stated as follows "If I increase the number of followers on my social media, then all these followers will use my product that I advertise on my social media.” 2 - The relationship is plausible. The independent metric has no direct correlation because the increase in social media followers is due to the content of your social networks – and not necessarily the likeability of your product. Step 4. Set the Validation Criteria Step 4. Set the Validation Criteria Step 4. Make sure that your hypothesis is built on realistic and measurable criteria. Let’s take a look at a couple of examples. First, you could use the ratio between active users and new users, where an index greater than 1 indicates an active use of the platform, and an index below 1 indicates that users are losing interest in the platform, as your confirmation criteria. Another criteria example could be the conversion rate (CR), which measures desired actions such as making a purchase, a ratio between those who see an object and those who wish to buy it. A high CR indicates that the product is desired or can be visualized as a desired category. A low CR may indicate that the products are not easy to sell. These examples show a strong correlation between their variables. Not all hypotheses are the same. They can be classified into null hypotheses or alternative hypotheses; the first being statements that negatively refute a hypothesis. For instance, “the number of active users will not increase if we implement a free delivery system" , and the latter aims to prove the validity of a hypothesis in a positive way, "a free delivery system will allow the number of active users to grow." “the number of active users will not increase if we implement a free delivery system" "a free delivery system will allow the number of active users to grow." Each hypothesis statement is different and may vary according to its subject, tone, and wording. It should be noted that a hypothesis statement should always have two or more variables and a connecting factor. Once you are clear on these points, you can focus on assigning priorities to each hypothesis. This will allow you to focus on the most impactful solutions in order to create the product roadmap, reduce your to-do list, and choose a product hypothesis to test. Step 5. Testing Hypothesis Step 5. Testing Hypothesis Step 5. Have you selected your final product hypothesis? Then let's put it to the test! It’s important for testing results to be as informative as possible. What’s more, the testing process is a continuous process that helps you learn more about your product and assumptions. Naturally, the most difficult part here is to decide which experiments to run. Remember, that you do not need to resort to software development, especially if you are running a start-up company and do not have resources to waste. Instead, you could use existing marketing tools to help you gather the required data. Landing Page Testing Landing Page Testing One of the popular ways to test hypotheses used by product managers around the world is creating a separate landing page for each experiment. Choose your target audience, and assign a distinct pain point to it. The more options you have, the better. But remember that there should be only one pain point assigned to each target - this will make debunking your testing results a lot easier. Create a landing page and a marketing campaign to fit each pain point. The landing page has to include a distinct design, pricing, and offer. You can change your variables depending on what you want to test. Place an opt-in form at the bottom of your landing page so that your target customers do not feel obligated to use your product or do not think that you want to obligate them. Instead, your goal is to collect honest feedback. You could also place a mini questionnaire inside your opt-in. Allocate the same budget to all the landing pages (hypotheses) you want to test. Compare the results using the same marketing funnel for all product hypotheses. A/B Testing Also known as split testing, A/B tests are a popular tool used by product managers and marketers to test all kinds of hypotheses. The point of the A/B test is to create two or more versions of the hypotheses, collect information about how users react to them, and choose the best option. For instance, if you want to implement free delivery for a specific category of products, you could test your hypothesis on whether it is better to choose the “beauty” or “clothing” category for the free shipping experiment via an A/B test. CustDev Interviews Last but definitely not least, simply ask your users what they want. CustDev interviews are a timeless way to strengthen your hypothesis test and a perfect step to start with. Yet, the instrument does not validate hypotheses quantitatively, if you find common patterns after 10+ interviews, you can further test it with quantitative methods, specify test details, or switch to another hypothesis without wasting time. CustDev is an in-depth interview with your clients, conducted online or in person in order to acquire as much feedback as possible. These kinds of interviews require a lot of effort to prepare but it is still an excellent way to quickly learn about your product assumptions. Conclusion Time to Take Action Conclusion Time to Take Action Conclusion Now that you have finished testing your hypothesis, it is time to evaluate the data. But before you do that, I recommend answering the following questions: How well was the test executed? Is my data clean and up-to-date? Do I need more time to continue the experiment? How well was the test executed? How well was the test executed? Is my data clean and up-to-date? Is my data clean and up-to-date? Do I need more time to continue the experiment? Do I need more time to continue the experiment? If you are not sure whether your test was successful, here is something you can keep in mind in order to validate your product hypothesis in the right way: Take maximum precautions when gathering data. Create analytical dashboards, receive consent from your clients to record interviews, and write down notes. Do not waste too much time on one experiment. You are likely to conduct a couple of experiments to test one hypothesis, but if your experiments are constantly not working out, then maybe you’d better change your hypothesis to get clearer results. Do not waste your time for too long. Choose the right target audience. The right audience is the foundation of your test, therefore, you must be specific about who you want to test your product on and avoid biases and/or irrelevant testing “just to validate your hypothesis” at all costs. Take maximum precautions when gathering data. Create analytical dashboards, receive consent from your clients to record interviews, and write down notes. Take maximum precautions when gathering data. Create analytical dashboards, receive consent from your clients to record interviews, and write down notes. Take maximum precautions when gathering data. Do not waste too much time on one experiment. You are likely to conduct a couple of experiments to test one hypothesis, but if your experiments are constantly not working out, then maybe you’d better change your hypothesis to get clearer results. Do not waste your time for too long. Do not waste too much time on one experiment. You are likely to conduct a couple of experiments to test one hypothesis, but if your experiments are constantly not working out, then maybe you’d better change your hypothesis to get clearer results. Do not waste your time for too long. Do not waste too much time on one experiment. Choose the right target audience. The right audience is the foundation of your test, therefore, you must be specific about who you want to test your product on and avoid biases and/or irrelevant testing “just to validate your hypothesis” at all costs. Choose the right target audience. The right audience is the foundation of your test, therefore, you must be specific about who you want to test your product on and avoid biases and/or irrelevant testing “just to validate your hypothesis” at all costs. Choose the right target audience.