Ditasko

How to Calculate the Right Sample Size for Your A/B Tests

Determining the appropriate sample size is one of the most critical yet often overlooked aspects of A/B testing. Too small a sample, and you risk inconclusive results or false positives. Too large, and you waste time and resources. This guide will walk you through calculating the right sample size for reliable tests.

Why Sample Size Matters

Proper sample size ensures:

  • Statistical power: Ability to detect real differences when they exist
  • Reliable results: Confidence that observed differences aren't due to chance
  • Efficient testing: Avoid running tests longer than necessary
  • Resource optimization: Don't waste traffic on inconclusive tests

Key Factors in Sample Size Calculation

Four primary factors determine required sample size:

1. Baseline Conversion Rate

The current conversion rate of your control version. Lower rates generally require larger samples.

2. Minimum Detectable Effect (MDE)

The smallest improvement you want to detect. Smaller effects require larger samples.

3. Statistical Significance Level

Typically 95% (α = 0.05). Higher confidence requires larger samples.

4. Statistical Power

Typically 80%. Higher power requires larger samples.

The Sample Size Formula

The standard formula for calculating sample size per variation is:

n = [ (Zα/2 + Zβ)² × p(1-p) ] / (p₁ - p₀)²

Where:
Zα/2 = Z-score for desired significance level (1.96 for 95%)
Zβ = Z-score for desired power (0.84 for 80%)
p₀ = baseline conversion rate
p₁ = expected conversion rate after improvement
p = (p₀ + p₁)/2

Practical Example

Let's say you have:

  • Baseline conversion rate (p₀): 5%
  • Want to detect a 10% relative improvement (p₁ = 5.5%)
  • 95% confidence (α = 0.05)
  • 80% power

Calculating:

p = (0.05 + 0.055)/2 = 0.0525
n = [(1.96 + 0.84)² × 0.0525(1-0.0525)] / (0.055 - 0.05)²
n = [7.84 × 0.0497] / 0.000025
n = 0.3896 / 0.000025
n ≈ 15,584 visitors per variation

So you'd need about 31,168 total visitors (15,584 in each variation) to reliably detect a 10% improvement from 5% to 5.5%.

Using Sample Size Calculators

While the math is important to understand, in practice you'll typically use calculators:

Online Sample Size Calculators

Tools like Evan's Awesome A/B Tools, Optimizely's Sample Size Calculator, or built-in calculators in testing platforms like VWO or Google Optimize make this easy.

Common Mistakes in Sample Size Calculation

Avoid these frequent errors:

  • Underestimating required sample size: Leads to inconclusive tests or false positives
  • Not accounting for traffic fluctuations: Weekdays vs weekends, seasonality
  • Changing goals mid-test: Switching primary metrics invalidates calculations
  • Ignoring unequal traffic splits: 80/20 splits need larger total samples than 50/50
  • Overestimating expected effect sizes: Most tests show smaller lifts than anticipated

Advanced Considerations

For more sophisticated testing programs:

Sequential Testing

Allows for smaller initial samples with the option to continue if results are promising but not yet significant.

Bayesian Approaches

Alternative methods that can sometimes reach conclusions with smaller samples, though interpretation differs.

Practical Tips

To implement sample size calculations effectively:

  1. Start with conservative effect size estimates (most tests show 5-15% lifts)
  2. Calculate sample size before starting any test
  3. Monitor actual vs expected conversion rates during the test
  4. Consider running tests for at least 1-2 full business cycles (weekly, monthly)
  5. Document your calculations and assumptions for future reference
"Proper sample size calculation is the difference between data-driven decisions and guessing with numbers."

By understanding and applying proper sample size calculations, you'll run more efficient, reliable A/B tests that produce actionable insights rather than inconclusive or misleading results.