Why Most A/B Tests Measure Outcomes Without Producing Insight
The hosts distinguish random cosmetic experiments from target-driven optimization and customer-problem research. They argue that the strongest tests begin with a customer insight and should explain why performance changed, not merely report a percentage lift.
- Random element changes are the weakest form of testing
- Chasing a numerical lift can cause wasteful iteration
- A percentage increase is an outcome rather than an insight
- Strong tests begin with a customer problem
- A successful test should explain why the result occurred
“if you're going to do an A B test, it should be a hypothesis driven test.”
“if an A B test works, you should be able to say the why.”