marketing
A/B Testing
Comparing two versions of something to see which performs better. The discipline of data-driven decisions.
Definition
A/B testing (split testing) runs two versions of a marketing element - headline, button color, email subject line, ad creative - against each other simultaneously to see which performs better. The discipline: change one variable at a time, run long enough to reach statistical significance, then implement the winner. Common mistake: calling winners too early. A test with 50 conversions is just noise; you typically need 300-500 conversions per variant for confidence.
Statistical significance in US A/B testing
A/B test results require enough data to be reliable. Statistical significance is the probability that observed difference is not due to chance. US convention. 95 percent confidence threshold (5 percent false positive rate). Minimum 300 to 500 conversions per variant for typical effect sizes. Test duration: 2 to 4 weeks minimum to capture day-of-week patterns. Tools that calculate significance: VWO, Optimizely, Convert, Google Analytics (with caveats). Common mistakes. Calling winners after 50 conversions when difference is noise. Stopping test the moment significance hits (peeking problem). Running test too long when business conditions change mid-test. The discipline: predefine sample size, do not peek, do not declare winners until significance and minimum sample both reached. Most US small business A/B tests fail not because variants are bad but because significance discipline is missing.
What to test and what not to test
Effective US A/B testing prioritization. High-impact tests. Headlines and value propositions (5 to 40 percent conversion lift potential). Primary CTA copy and design (5 to 20 percent). Form fields and length (5 to 30 percent). Hero images and video (5 to 25 percent). Pricing presentation (5 to 25 percent). Low-impact tests rarely worth running. Button colors (less than 5 percent lift, often noise). Header font choices (rarely significant). Minor wording changes in body copy (small effect). Tools that help prioritize: ICE scoring (Impact, Confidence, Ease), value-versus-effort matrix. US small businesses with limited traffic should focus on 3 to 6 high-impact tests per year rather than 30+ small tests that never reach significance. The discipline of choosing fewer, bigger tests produces more learning than scattered small tests.
Testing without enough traffic
Many US small businesses lack the traffic for traditional A/B testing (need 600 to 1000 conversions to test, often have 50 to 200 per month). Adaptations for low-traffic businesses. Qualitative testing: customer interviews and usability sessions reveal directional insights without statistical proof. Run 5 to 10 user interviews before any major design change. Sequential testing: test one variant for a full month, baseline the result, then test alternative the next month. Less rigorous than parallel A/B but workable when traffic is limited. Multivariate-impossible: focus on big swings rather than incremental tweaks. Change the entire offer or headline rather than testing button colors. Tools that help: Hotjar (heatmaps and recordings), UserTesting (recorded sessions), Lookback (live user research). For US small businesses under 2K monthly visitors, qualitative research often beats quantitative testing.
Common US A/B testing mistakes
Five errors that produce false conclusions. One, testing multiple variables simultaneously without proper multivariate framework. Cannot isolate which change drove the result. Two, peeking at results and stopping when significance temporarily hits. Inflates false positive rate dramatically. Three, ignoring external factors (seasonality, paid ad changes, news events) that affect both variants but unevenly. Four, declaring winners on small samples that look significant but lack power. Differences disappear when scaled. Five, applying learnings from one segment or channel to all segments. Mobile and desktop behavior differs; B2B and B2C behavior differs. Each mistake produces apparent wins that do not replicate. The discipline of methodological rigor matters more than testing volume; a single well-designed test produces more learning than 10 poorly-designed tests.
FAQ
How long should I run an A/B test?
Minimum 2 weeks to capture weekly patterns; 4 weeks better for B2B with longer cycles. Maximum 8 weeks before external factors (season, competitor moves, your other changes) confound results. Within that window, run until statistical significance reaches 95 percent confidence AND minimum sample size hits (typically 300 to 500 conversions per variant). If significance does not reach 95 percent after 6 to 8 weeks, the effect is probably too small to be detected with your traffic level; move to next test.
Can I A/B test on a low-traffic site?
Difficult with traditional A/B testing methodology; consider alternatives. US small business sites with under 2K monthly visitors usually cannot reach statistical significance in reasonable time. Alternatives that produce learning. Qualitative user testing (5 to 10 interviews, faster directional insight). Heatmap and session recording analysis (Hotjar, FullStory). Sequential testing (one variant for a month, baseline result, then alternative the next month). Larger before-and-after changes (modify the entire offer rather than testing micro variations). The right approach depends on traffic level and decision time horizon; pure quantitative A/B testing requires meaningful traffic.
What is the minimum sample size for a valid A/B test?
Depends on baseline conversion rate, expected lift, and confidence level. Quick rule for typical US conversion rates. Detecting 20+ percent relative lift on 5 percent baseline conversion: roughly 6,000 visitors per variant at 95 percent confidence. Detecting 50 percent lift on same baseline: roughly 1,000 visitors per variant. Detecting 10 percent lift on same baseline: roughly 25,000 visitors per variant. Use online A/B test calculators (Optimizely, VWO, Evan Miller) to compute exact sample requirements for your specific conversion rate and effect size expectations. Setting up the math before testing prevents calling false winners on insufficient data.
Should I test mobile and desktop together or separately?
Separately, ideally. Mobile and desktop user behavior differs significantly in US data. Mobile typically has lower conversion rates, different click patterns, different attention spans. A change that helps desktop may hurt mobile or vice versa. Segment your A/B tests by device when possible; some testing tools (Optimizely, VWO) support automatic device segmentation. If traffic does not support separate tests, prioritize mobile because mobile typically dominates traffic for US consumer sites and matters more for younger demographics.
What if my A/B test shows no difference?
Three interpretations. One, the variants are genuinely equivalent for your audience; the change does not matter. Move on to bigger swings. Two, the test lacked statistical power; effect exists but is too small to detect at your traffic level. Either accept the uncertainty or run a larger test. Three, the test is flawed (mixing audience segments, contaminated by external factors, technically broken). Audit the setup before drawing conclusions. Many US A/B tests show no significant difference because the variants tested were too similar; bigger swings produce more learning than micro tweaks.
In your business
- →Change one variable at a time - otherwise you can't isolate what worked
- →Wait for statistical significance - rule of thumb: 300+ conversions per variant
- →Test high-traffic, high-impact elements first - headline, CTA, hero image