A/B Test Calculator

Calculate statistical significance for A/B tests and determine required sample sizes

Test Type

Conversion Rate Data

Variant A (Control)

Rate: -

Variant B (Treatment)

Rate: -

Test Parameters

Help

How to Use
  1. Choose your metric type (conversion rate or continuous)
  2. Enter data for both variants (A and B)
  3. Set your confidence level (typically 95%)
  4. Choose one-tailed or two-tailed test
  5. Click "Calculate Results" to see analysis
  6. Review statistical significance and confidence intervals
Understanding Results
  • P-value: Probability of observing this difference by chance; p < 0.05 indicates significance
  • Confidence Interval: Range where the true difference likely lies; narrower is more precise
  • Effect Size: Magnitude of difference; helps assess practical significance
  • MDE: Minimum detectable effect - smallest difference you can reliably detect
Test Types
  • Conversion Rate: Use for binary outcomes (clicked/didn't click, converted/didn't convert)
  • Continuous: Use for numeric metrics (revenue, time on page, items purchased)
  • Two-tailed: Tests if B is different from A (higher OR lower) - more conservative
  • One-tailed: Tests if B is specifically higher (or lower) than A - use only with strong prior hypothesis
Sample Size Guidelines
  • Larger samples detect smaller differences and give more precise estimates
  • For conversion rates, aim for 100+ conversions per variant minimum
  • Plan sample size in advance based on expected effect size and desired power
  • Don't stop early just because results are significant (inflates error rates)