A/B Tests Worth Running: A List for Product Designers
The 'let's test it' culture often becomes an excuse for a lack of strategic thinking. Not every idea warrants an A/B test, especially for small startups with relatively low traffic where achieving statistical significance can take months. So, what *is* worth testing?
Information Architecture vs. Cosmetics
Testing button color (from green to blue) is a mythical example from the early 2000s. In reality, differences are usually within statistical error. A/B tests that yield double-digit conversion uplifts typically involve changes to structure or the Value Proposition.
Here are 3 types of A/B tests with the highest ROI:
- The order of sections on a Landing Page. (E.g., Hero -> Social Proof -> Pricing vs. Hero -> Features -> Social Proof).
- Microcopy on Call-to-Actions. Changing 'Sign Up' to 'Start Free Trial' completely alters the perceived risk for the user.
- Form length (Reducing fields vs. multi-step forms).
The Peeking Error
The biggest sin in CRO is stopping tests as soon as we see a p-value of 0.05 on the third day of an experiment.
Traffic distribution throughout the week is not uniform. People purchase differently on Mondays at 9:00 AM than on Saturdays at 10:00 PM. An A/B test must run through full business cycles (typically 2-4 weeks) for the results to be reliable. Establish your MDE (Minimum Detectable Effect) before starting and stick to the plan.