Talking to founders, we often encourage a mindset of experimentation and testing, it’s a natural part of any business and forms the foundations of innovation.
But I’ve realised that startups don’t really know how to start or where to begin. It’s easy to talk about the idea of doing lots of experiments, but execution is a whole different matter, experiments take time, they take man power, they take investment, thats a lot to ask from founders already eating ramen and trying to find time for VC pitches in between trying to make ends meet.
Most of the time, founders get really nervous at the mention of doing experiments. They’ve already put so much into building product and trying to build a plane and fly it, the mere thought of getting feedback that might provide evidence they’ve been going down the wrong path is enough for many founders to opt, instead, to putting their head in the sand.
But doing experiments doesn’t necessarily mean taking months, giving away huge discounts, implementing massive tech tools or sucking up hours and hours of manhours.
In fact, quite the opposite, experiments are designed to give feedback and improve clarity. So if the question is simple enough, the sample size, or testing can also be smaller and simpler.
So a better way to look at experiments is to consider that the duration, complexity and effort invested into an experiment should be proportional to the impact the answer would have, the risk associated with getting the answer wrong, and the clarity of the answer.
So, if the impact is massive, the risks of getting it wrong are huge, and the responses vague, then extensive and detailed testing is needed.
On the other hand, if the risks are low, impact low, and signal is clear, it might also not be worth testing at all. some times the answers are obvious.
So, with this in mind, a better way to think of experiments is that they should be progressive you want to start with something small and manual and see if the signal is clear. For example, we want to test a new marketing approach.
You could choose to launch it via an email to the entire database. or you could call 5 target candidates and see if you get the response you want. if 5/5 give you exactly what you were looking for, then you have already de-risked the experiment, if 3/5 give you the response you want, then you’d look for a way to expand the experiment.
In this way, optimising something like a landing page doesn’t require crazy tracking technology at this early stage… instead all you need to do is have two landing page URLs sent to two comparable but different traffic sources (e.g. by splitting an email database in two, with group A receiving landing page A and group B receiving landing page B)