A/B Test Significance Calculator
Compare conversion rates between a baseline and one or more variants using a two-proportion z-test. Get a p-value, confidence interval, lift, and a clear significance call β or run a full A/B/n comparison with Bonferroni correction and automatic winner detection.
Background
Every comparison here is a standard two-proportion z-test: baseline vs. one variant. A result can be statistically significant without being practically meaningful β a tiny lift can be "significant" with a huge sample, while a real lift can look inconclusive with a small one. Testing many variants against one baseline also inflates the odds of a false positive, which is exactly what the Bonferroni option corrects for.
How to use this calculator
- Choose Compare Two Variants for a simple A/B test β enter visitors and conversions for baseline and variant, pick your significance level and hypothesis.
- Choose Compare Multiple Variants for A/B/n testing β add up to 8 variants, pick a baseline, and optionally apply Bonferroni correction.
- Click Calculate to see the p-value, confidence interval, lift, a significance gauge, and a full step-by-step explanation.
How significance testing works
A two-proportion z-test asks: if there were truly no difference between baseline and variant, how surprising would this observed gap be? The p-value is the answer, expressed as a probability.
Statistical significance (p β€ Ξ±) means the difference is unlikely to be due to chance. It does not mean the lift is large enough to matter for your business β that's practical significance, and it's a separate judgment call.
Testing many variants against one baseline multiplies your chances of a false positive. The Bonferroni correction tightens the bar (Ξ±β² = Ξ± Γ· number of comparisons) to compensate.
A two-sided test asks "is there any difference at all?" A one-sided test asks a directional question ("is the variant better?") and should only be used if that decision rule was set before looking at the data.
Small samples reduce statistical power β a real, meaningful lift can easily fail to reach significance if too few visitors were tested, which is why a "not significant" result never proves there's no difference.
Formula & Equations Used
Conversion rate: p = x Γ· n
Pooled rate: pΜ = (xβ + xβ) Γ· (nβ + nβ)
Standard error (pooled): SE = β(pΜ(1βpΜ)(1/nβ + 1/nβ))
z-statistic: z = (pβ β pβ) Γ· SE
Confidence interval for Ξ: Ξ Β± z* Β· β(pβ(1βpβ)/nβ + pβ(1βpβ)/nβ)
Bonferroni-adjusted Ξ±: Ξ±β² = Ξ± Γ· (number of comparisons)
Example Problems & Step-by-Step Solutions
Example 1 — Classic A/B (two-sided)
Baseline: 10,000 visitors, 400 conversions (4.00%). Variant: 10,000 visitors, 520 conversions (5.20%). Ξ± = 0.05, two-sided.
Step: Ξ = 1.20 pp. Pooled rate pΜ = 4.60%. z β 4.05, p β 5.1Γ10β»β΅.
Result: p β€ 0.05, so this is significant β and the 95% CI for Ξ (0.62 pp to 1.78 pp) stays entirely above zero.
Example 2 — One-sided "B > A" decision rule
Baseline: 5,000 visitors, 150 conversions (3.00%). Variant: 5,000 visitors, 210 conversions (4.20%). Ξ± = 0.05, one-sided (variant > baseline).
Step: Ξ = 1.20 pp (+40% relative). z β 3.22, one-sided p β 0.00064.
Result: significant evidence that the variant improves on baseline, under a pre-set directional rule.
Example 3 — A promising lift that isn't significant
Baseline: 500 visitors, 10 conversions (2.00%). Variant: 500 visitors, 12 conversions (2.40%). Ξ± = 0.05, two-sided.
Step: Ξ = 0.40 pp (+20% relative). z β 0.43, p β 0.67.
Result: a 20% relative lift sounds exciting, but with this little data it's statistically indistinguishable from noise β the CI (β1.42 pp to 2.22 pp) crosses zero.
Example 4 — A/B/C with Bonferroni
Baseline A: 12,000/540 (4.50%). B: 12,000/660 (5.50%). C: 12,000/690 (5.75%). Ξ± = 0.05, two-sided, Bonferroni.
Step: Ξ±β² = 0.05 Γ· 2 = 0.025. B vs A: p β 0.00038. C vs A: p β 0.0000113.
Result: both variants clear the stricter Ξ±β² = 0.025 bar β C has the strongest evidence and is declared the winner.
Example 5 — When the baseline wins
Baseline A: 8,000/800 (10.00%). B: 8,000/680 (8.50%). C: 8,000/810 (10.13%). Ξ± = 0.05, two-sided.
Step: B vs A: Ξ = β1.5 pp, p β 0.0011 (significant, but worse). C vs A: p β 0.79 (not significant).
Result: no variant improves on the baseline, and B is significantly worse β the calculator flags this as "Control wins."
Frequently Asked Questions
What does "statistically significant" mean?
It means the observed difference is unlikely under the assumption that there's truly no difference, at your chosen significance level (Ξ±). It's a statement about confidence, not about size.
Is it okay to compare many variants (A/B/n)?
You can, but each additional comparison raises the odds that at least one looks "significant" purely by chance. Bonferroni correction (Ξ±β² = Ξ± Γ· comparisons) is a simple, conservative way to compensate.
Should I use a two-sided or one-sided test?
Default to two-sided. Only use one-sided if your decision rule ("we'll ship only if the variant beats baseline") was fixed before you looked at any results β choosing it after seeing the data inflates false positives.
Does significance guarantee the variant is actually better?
No. Significance is about confidence in a difference existing, not about whether that difference is big enough to matter. Always check the lift, the confidence interval, and any guardrail metrics before shipping.
Why did the calculator declare "Control wins" instead of a variant?
That happens when no variant significantly beats the baseline, but at least one is significantly worse β a genuinely useful outcome to flag, since it means the safest choice is to keep what you have.
What's the continuity correction (Yates) for?
It's a small adjustment that makes the z-test more conservative, mainly useful for smaller sample sizes where the normal approximation is rougher. It never increases significance, only reduces it.
All calculators