‍

More and more organizations test their business ideas before implementing them. The best ones perform a mix of experiments to prove that their ideas have legs. They ask two fundamental questions to design the ideal mix of experiments:

1. Speed: How quickly does an experiment produce insights?
2. Strength: How strong is the evidence produced by an experiment?

Easy vs Hard To Set Up Experiments

With any new business experiment you need to ask yourself how quickly you can get started and how quickly it produces insights. For example, an interview series with potential customers or partners can be set up fairly quickly. Launching a landing page and driving traffic to it can be done with even greater speed. You’ll generate insights quickly. A technology prototype on the other hand will take far more time to design and test. Such a prototype might gather a good understanding of user behaviour, but it will require more time to generate insights.

Weak or Strong Evidence

Not all evidence is equal. Interviews and surveys are fast to set up, but we all know that what (potential) customers and partners say and do are two different things. On the other hand, a simulated sales--where the customer doesn’t even know that she is part of an experiment--can get you pretty close to a real world purchasing situation. The first experiment produces valuable, but weak evidence. The latter produces strong evidence, but requires that you already have a good understanding of what customers might be interested in.

From Quick Evidence to Strong Evidence

As a rule of thumb you should start with quick and cheap experiments at the early stages of testing a new business idea. At this exploratory stage there’s a lot of uncertainty and you don’t yet know in which direction you should head. The strength of the evidence is not yet crucial. You’re just trying to better understand potential customers, their jobs, pains, and gains, and their ability and willingness to pay.

Once you know where you’re heading and you’re beyond exploration you need to validate that you’re on the right track. For this phase use longer and more solid experiments to produce strong evidence. Strong evidence, like the data from simulated sales, allows you to predict future customer behaviour with confidence. In general, the evidence gets stronger the closer you get to a real world buying example. In some industries, like pharmaceuticals, it’s not possible to simulate sales, but you need to be creative to get as close as you can.

What’s your mix of experiments for exploration and validation?