What does “statistically significant” mean in an A/B test?

It means the observed difference is unlikely to be due to random chance at your chosen significance level (alpha).

What inputs do I need for this calculator?

You need visitors (or trials) and conversions for each variant, then choose a baseline to compare against.

Which statistical test does this use?

It uses a two-proportion z-test for each variant vs the chosen baseline.

Should I adjust for multiple comparisons in A/B/n?

When comparing many variants vs a baseline, the chance of false positives increases. A conservative option is Bonferroni adjustment (alpha divided by the number of comparisons).

All calculators

A/B Test Significance Calculator

Compare conversion rates across variants. Pick a baseline (usually Control A), and we’ll compute p-value, confidence interval, lift, and a significance call for each variant vs the baseline using a two-proportion z-test. Supports A/B/n (multiple variants) — each variant is compared to your chosen baseline.

Background

Each comparison is a standard two-proportion z-test (baseline vs one variant). For true A/B/n experiments, repeated comparisons can inflate false positives (multiple testing). This tool is ideal for fast checks and clear reporting.