What are the principles of selecting an experiment?
Before jumping to an experiment, it is important to consider the key principles of rapid experimentation when testing a new business idea.
- Go cheap and fast early on in your journey - Don’t spend a lot of money if possible early on. You are only beginning to understand the problem space, so you don’t want to spend money when you can learn for free. Also try to move quickly to learn fast, instead of slow and perfect.
Quite often, I am asked “how many experiments should I run”? This is a very hard question to answer as it depends on so many variables. But a general rule of thumb is 12 experiments in 12 weeks. 1 experiment a week is a good pace to keep up momentum and 12 data points is typically a good checkpoint to come back and reassess your business model. - Increase the strength of evidence with multiple experiments for the same hypothesis - Don’t hesitate to run multiple experiments for the same hypothesis. Rarely do we witness a team that only runs one experiment and uncovers a multimillion dollar opportunity. The idea here is to not get too excited or too depressed after running one experiment. Give yourself permission to run multiple to understand if you have genuinely validated your hypothesis.
Remember, a typical business model may have several critical hypotheses you need to test and for each hypothesis, you may need to run multiple experiments to validate it. Refer to our previous blog post on Assumptions Mapping to learn how to define your hypotheses and determine which ones to run first. - Always pick the experiment that produces the strongest evidence, given your constraints - Not every experiment applies to every business. B2B differs from B2C which differs from B2G. A 100 year old corporation’s brand is much more important than a 100 hour old startup’s brand. Pick an experiment that produces evidence, but don’t risk it all. Make small bets that are safe to fail.
When working with corporate innovators, the phrase I most often hear is “we can’t do that”. Innovators often feel hamstrung when working in heavily regulated industries. Many may choose to bypass testing certain hypotheses because of the constraints. This is not something I would recommend. If you really cannot test a hypothesis, the last resort would be to consider pivoting your business model. Remember, a testable idea is always better than a good idea. - Reduce uncertainty as much as you can before you build anything - In this day and age you can learn quite a bit without building anything at all. Deferring your build as long as possible, because it is often the most expensive way to learn.
In the Experiment Library we’ve compiled a list of creative ways to demonstrate a prototype without building out your final product. Experiments such as Wizard of Oz, Clickable Prototype and Single Feature MVP.
Discovery or validation, what’s the difference?
Pulling inspiration from Steve Blank’s 4 Steps to the Epiphany, it is quite obvious that we framed the experiment library in a similar fashion. One of the biggest differences however is that instead of “customer discovery” and “customer validation” we decided to test the entire business model. Discovery experiments are often open ended and directional, whereas the Validation experiments have more of a true value exchange. Most teams start with discovery since the validation experiments cost more and take longer.
Here is our one page cheat sheet to the Experiment Library.
You will learn about templates and more by signing up for our master workshop on Testing Business Ideas here.
