Entrepreneurs and corporate innovators alike often struggle to figure out when a hypothesis is validated. Because it isn’t as simple as we’d like to believe.
The question “has your hypothesis been validated?” by leadership is often a point of contention.
Teams do not see a clear validated signal, because the insights gained from the evidence are on a spectrum. In fact, this spectrum reflects our two themes of experimentation - discovery and validation - from the book Testing Business Ideas.
Discovery is much more about open-ended, directional experiments where you are trying to go from no evidence to light evidence. This is evidence of what people say, based on opinions and experiments in a lab context where customers know it's a test.
Validation experiments usually come much later, when you are going from some evidence to strong evidence. This is evidence of what people actually do, based on facts and figures from the real world. The goal is to get irrefutable evidence from the market. For example, sales from a pop-up store or mock sales. Where customers believe they are actually purchasing a product or service.
Alex Osterwalder and Yves Pigneur created the Innovation Project Scorecard. As a simple way to measure and track how much risk you have reduced in your business idea. Teams measure their desirability, feasibility, viability, and adaptability risk from 0 to 5. Based on the type and amount of evidence they collect.
Measuring Risk
The scoring system is base on two components to assess how much risk you have reduced in your business idea. Ie, how much closer you are to validating your hypothesis.
- Type of evidence. This determines how strong your evidence is and is the primary indicator of how you should score yourself.
- The number of data points you have collected for that experiment. This should help you determine how confident you are in the insights captured.
Strength of evidence
Not all evidence is of equal strength. It can vary from very weak/light evidence to very strong and even irrefutable evidence. This range is determined by the type of experiment you conduct and the evidence it produces.
You can split evidence into two broad buckets. One is always stronger than the other: what people say versus what people do. For example, interviews are a fast and cheap way to capture initial customer insights. But, interviews only reveal what customers say, and are hence not very reliable. An experiment where you can observe what customers actually do is always stronger. The same applies for when customers share opinions versus facts in an interview. You get facts when you ask “when is the last time you ....”. You get opinions when you ask “would you...?”, "what do you think?", or "do you like ....?" questions.
Another way to determine the strength of evidence is to look at the investment made by a test subject. For example, it's common practice to launch landing pages for a future product. The experiment is to measure how many people convert to provide their email for the launch date. This is good evidence of interest, but very weak/light. When Tesla launched their first car, it also created a landing page. But, they asked for a deposit to reserve a car, rather than an email. That is stronger evidence. They repeated this experiment with the Model 3 and gathered 325,000 deposits in one week.
Lastly, the experiment itself influences the strength of evidence.
Customers who know you are testing behave differently than those who don't. The more the scenario is real rather than a "lab context", the stronger the evidence. Test subjects will interact differently with a gadget in a home or lab setting. Logitech learned this the hard way when they lost $100 million with Google TV in 2011. The extensive lab experiments didn't predict real-world failure.1
Generally, experiments that give you stronger evidence take longer to set up and are more costly. This is why we always recommend starting with lighter evidence experiments that are quick and cheap. They allow you to “get out of the building” and start testing straight away. Even if you collect weak evidence, it's still better than no evidence at all.
This brings us to the second point, how much evidence you collect matters.
Number of data points
The number of data points for an experiment will impact how confident you feel about the results. For example, 60% of customers interviewed said they have encountered the pain you are addressing. The relevance and strength of this finding will depend on if you have interviewed 5 people or 100. , you can feel much more confident about the results from the latter. That said, customer interviews is weak evidence, since it is still what people say. This means your evidence score will never exceed a 2, no matter how many interviews you conduct. While this is the case, you can still feel a lot more confident that you are working on a real problem after speaking to 100 people. And that should be the aim of your first experiments.
The same is also true for experiments generating stronger evidence, like a pre-sale. You can feel a lot more confident about your idea if 50 people paid a deposit to buy it versus 5. While this is strong evidence, it would still fall short of the ‘irrefutable evidence’ benchmark. As you are still only asking people to pay a deposit rather than outright buying the product. One way to generate irrefutable evidence is to do a covert sale. Customers will believe they are buying the real thing, and hence give you a genuine response. You will gain a new level of understanding without having to build the final product.
Whether you have validated a hypothesis is nuance. At the end of the day, it is a conversation between the team members and a judgment call only you and your team can make. Using the Innovation Scorecard will guide your team to have concrete and evidence based conversations.
Leaders can help these discussions, not by asking whether your hypotheses have been validated but rather, by asking teams what evidence has been collected and how confident they feel about that evidence.
You can download the scorecard here.
Learn more about evidence-based decision-making by signing up for our master workshop on Testing Business Ideas here.
Link: https://www.cnet.com/news/logitech-confesses-to-gigantic-mistake-with-google-tv/
