A/B Testing for Retail: When It Works and When It Doesn’t

A/B testing is the gold standard for measuring causal impact. In retail it can work — but the constraints are different from digital-only businesses. Smaller sample sizes, slower iteration, and external noise all complicate experiments. Here is how to do it right.

Physical store testing

Treat stores as the unit of test rather than customers. Run matched-pair tests where treatment and control stores have similar baselines. Allow at least 8 weeks of data per test to ride through weekly noise.

Digital retail testing

E-commerce A/B testing is mature: use Optimizely, VWO, or in-house frameworks. Be careful with novelty effects and ensure statistical significance before declaring winners.

Common pitfalls

Sample size too small, calling tests too early, ignoring multiple comparison problems, not controlling for seasonality, peeking at results daily.

When not to A/B test

Brand changes, store layout overhauls, and category resets are too disruptive for clean A/B tests. Use before/after with control stores instead. Reserve A/B tests for cleanly isolatable changes.