Return on Testing – the impact of rapid testing culture

Startup culture encourages rapid, high volume testing and experimentation, with a few that the more you try the more you’ll learn and eventually succeed, but this can result in a lot of wastage. It’s not that testing and experimentation is wrong. Its more that it needs to firstly be directed and focused towards a meaningful end goal, the results of the experiment need to be impactful and provide enough data to be useful, and the cost of undertaking the experiments is sufficiently low in terms of time, effort and money. 

Coming up with wildly sophisticated complex experiments are all well and good on paper, but if the results are marginal, an additional 1 or 2 users here and there, there are more efficient ways to make an impact.

Culture means that we’re driven to run experiments that are smaller and more detailed that give lots of small wins, this makes us look good, and gives us a small dopamine hit that we’re making an impact. Running a large volume of small tests makes you look productive, even if it’s not moving the needle, so the more company politics your team is exposed to, the more likely they are to take safer bets. These Safe bets might be things like – button colours or Emojis in email subject lines.

Its therefore more impactful, to take more deliberate swings, and ensure that all of these swings are striving for  maximal impact. 

“Outsized returns often come from betting against conventional wisdom, and conventional wisdom is usually right. Given a ten percent chance of a 100 times payoff, you should take that bet every time. But you’re still going to be wrong nine times out of ten. We all know that if you swing for the fences, you’re going to strike out a lot, but you’re also going to hit some home runs.” – Jeff Bezos 

(Summary of – How to Calculate the Impact (ROI) of Testing — Reforge)

Leave a Reply

Your email address will not be published. Required fields are marked *