What is statistical significance in A/B testing?

Statistical significance tells you whether the difference between your control and variant is likely real or just noise. When a result is significant at 95%, it means there is less than a 5% chance the observed difference occurred by random variation alone. It does not tell you the size of the effect or whether the effect matters for your business. Always pair significance with the actual uplift percentage.

Can I check significance while the test is still running?

You can, but doing so inflates your false positive rate. This is called peeking. If you check daily and stop as soon as you see significance, your real error rate can be 20-30% instead of 5%. Either commit to a fixed sample size before starting, or use a sequential testing method designed for continuous monitoring. This tool assumes a fixed-horizon test.

What is statistical power and why does it matter?

Power is the probability that your test detects a real effect when one exists. At 80% power, you have a 20% chance of missing a real improvement (a false negative). At 90% power, that drops to 10%, but you need more visitors. Most teams start with 80% power. Increase to 90% when the cost of missing a real winner is high.

What does the p-value actually mean?

The p-value is the probability of seeing a result this extreme (or more extreme) if there were truly no difference between control and variant. A p-value of 0.03 means there is a 3% chance the observed gap is pure noise. It does not mean there is a 97% chance the variant is better. Common misreadings of p-values lead to overconfidence in test results.

How do I handle multiple variants (A/B/C tests)?

When you test three or more variants against one control, you need to correct for multiple comparisons. Without correction, each comparison has its own false positive risk, and those risks stack. Apply a Bonferroni correction by dividing your significance level by the number of comparisons. For example, with three comparisons at 95% confidence, test each at 98.3% (alpha = 0.05/3 = 0.0167). This tool handles two-variant tests. Run it separately for each variant-vs-control pair and apply the correction manually.

What if my test shows a negative result?

A negative result means the variant performed worse than the control. If the difference is statistically significant, it is a valid finding: the change hurt performance. Do not launch it. If the difference is not significant, the test is inconclusive. You cannot conclude the variant is the same as the control. You may need more data or a larger change to detect a difference.

A/B Test Statistical Significance Calculator

Run a pooled two-proportion z-test on finished experiment counts, read off the p-value, or flip to sample-size mode for absolute percentage-point lifts at 80% or 90% power.

100% client-side. Your inputs stay in this browser.

Switch between evaluating live control vs variant counts and planning traffic for an absolute percentage-point lift before you ship.

ControlVisitorsConversions

VariantVisitorsConversions

Confidence level

Two-sided test at 95% confidence

Not significant yet

p = 0.15

You need more data to reach 95% confidence.

Control CVR

5.00%

Variant CVR

6.50%

Relative uplift

+30.0%

Z-score

1.441

Critical |z| ≥ 1.96

How this tool works

The calculator offers two modes. In Test Results mode, it compares the conversion rates of your control group and variant group. You enter the number of visitors and conversions for each group, then pick a confidence level (90%, 95%, or 99%). The tool pools both groups to estimate a shared conversion rate, calculates a standard error, and produces a z-score. If the z-score exceeds the threshold for your chosen confidence level (1.645 for 90%, 1.960 for 95%, 2.576 for 99%), the difference is statistically significant. The tool also shows the p-value and the relative uplift in percentage terms. In Sample Size mode, the process runs in reverse. You enter your current baseline conversion rate, the smallest improvement you want to detect (minimum detectable effect), your confidence level, and your desired statistical power. The tool returns the number of visitors you need per variant and the total across both variants. This lets you plan before you run the test rather than checking results mid-experiment, which inflates false positive rates.

Worked example

A landing page test ran for two weeks. The control had 1,000 visitors and 50 conversions (5.0%). The variant had 1,000 visitors and 75 conversions (7.5%). With a 95% confidence level, the pooled conversion rate is 6.25%, the standard error is 0.01082, and the z-score is 2.31. Since 2.31 exceeds the 1.96 critical value, the result is statistically significant. The p-value is approximately 0.021, and the relative uplift is 50%. In Sample Size mode, detecting a 1 percentage point lift from a 5% baseline at 95% confidence and 80% power requires approximately 7,700 visitors per variant (15,400 total), illustrating how much traffic even a modest improvement target demands.

Frequently asked questions

What is statistical significance in A/B testing?
Statistical significance tells you whether the difference between your control and variant is likely real or just noise. When a result is significant at 95%, it means there is less than a 5% chance the observed difference occurred by random variation alone. It does not tell you the size of the effect or whether the effect matters for your business. Always pair significance with the actual uplift percentage.
Can I check significance while the test is still running?
You can, but doing so inflates your false positive rate. This is called peeking. If you check daily and stop as soon as you see significance, your real error rate can be 20-30% instead of 5%. Either commit to a fixed sample size before starting, or use a sequential testing method designed for continuous monitoring. This tool assumes a fixed-horizon test.
What is statistical power and why does it matter?
Power is the probability that your test detects a real effect when one exists. At 80% power, you have a 20% chance of missing a real improvement (a false negative). At 90% power, that drops to 10%, but you need more visitors. Most teams start with 80% power. Increase to 90% when the cost of missing a real winner is high.
What does the p-value actually mean?
The p-value is the probability of seeing a result this extreme (or more extreme) if there were truly no difference between control and variant. A p-value of 0.03 means there is a 3% chance the observed gap is pure noise. It does not mean there is a 97% chance the variant is better. Common misreadings of p-values lead to overconfidence in test results.
How do I handle multiple variants (A/B/C tests)?
When you test three or more variants against one control, you need to correct for multiple comparisons. Without correction, each comparison has its own false positive risk, and those risks stack. Apply a Bonferroni correction by dividing your significance level by the number of comparisons. For example, with three comparisons at 95% confidence, test each at 98.3% (alpha = 0.05/3 = 0.0167). This tool handles two-variant tests. Run it separately for each variant-vs-control pair and apply the correction manually.
What if my test shows a negative result?
A negative result means the variant performed worse than the control. If the difference is statistically significant, it is a valid finding: the change hurt performance. Do not launch it. If the difference is not significant, the test is inconclusive. You cannot conclude the variant is the same as the control. You may need more data or a larger change to detect a difference.