A/B Test Sample Size Calculator (2026)

How does the A/B test sample size calculator work?

Sample size in A/B testing determines how many observations are needed to detect a true difference between variations with a specified level of confidence. The calculator uses the standard two-proportion z-test formula for sample size. You provide four parameters: your baseline conversion rate, the minimum detectable effect (the smallest relative improvement worth detecting), the statistical significance level (confidence that the result is not due to chance), and statistical power (the probability of detecting a real effect when one exists). The tool converts your significance and power levels into z-scores, computes the pooled variance of the two conversion rates, and outputs the visitors per variant needed to detect your specified lift. It also calculates test duration by dividing the per-variant sample size by your daily traffic per variant. A curve chart shows how sample size changes as you adjust the MDE, so you can see the trade-off between sensitivity and test duration.

When should you calculate sample size before testing?

Calculate sample size before launching any A/B test where business decisions depend on the outcome. Pre-test calculation prevents two common mistakes: stopping a test too early when you see a promising result, and running a test indefinitely without enough traffic to reach significance. If your test requires 50,000 visitors per variant but your site only gets 1,000 visitors per week, you will know before launch that the test needs 100 weeks to conclude. That insight lets you either widen the minimum detectable effect to reduce the required sample, test a higher-traffic page, or choose a different optimization lever. Skipping the calculation and checking significance repeatedly as data arrives inflates your false positive rate well above the nominal 5% threshold. Every peek adds another opportunity to stop on noise rather than signal.

What factors affect required A/B test sample size?

Four inputs control the required sample size. Baseline conversion rate matters because detecting a 10% relative lift from a 50% baseline (5 percentage points) requires far fewer observations than the same relative lift from a 1% baseline (0.1 percentage points). The minimum detectable effect sets sensitivity. Detecting a 5% relative improvement demands roughly four times the sample of a 10% improvement. Statistical significance at the 95% confidence level means there is only a 5% chance the observed difference occurred by random chance. Raising confidence to 99% increases sample requirements. Statistical power is the probability of detecting a real effect when one truly exists. Standard practice uses 80% power, meaning you accept a 20% chance of missing a real improvement. Increasing power to 90% raises the sample size by about 30%. Daily traffic per variant determines test duration once you know the required sample. If you need 10,000 visitors per variant and each variant receives 500 visitors per day, the test runs for 20 days.

How do you interpret sample size calculation results?

The calculator returns three numbers: visitors per variant, total visitors across both variants, and estimated test duration. Visitors per variant is the count each of your two groups (control and treatment) must reach before you evaluate statistical significance. Total visitors is simply twice that figure, since you split traffic evenly. Test duration translates the sample requirement into calendar time by dividing visitors per variant by your daily traffic per variant. If the duration exceeds your planning horizon or patience threshold, you have three options: accept a larger minimum detectable effect (which lowers the required sample), increase confidence in a smaller effect by running longer, or test a different page with higher baseline traffic. The MDE sensitivity curve on the right shows how sample size and duration scale as you adjust the effect size. Small changes to MDE produce large changes in required sample, so the curve helps you find the practical boundary between statistical rigor and business constraints.

What is statistical significance in A/B testing?

Statistical significance measures the probability that an observed difference between variants arose purely from random variation rather than a real underlying effect. A result is statistically significant at the 95% confidence level when the probability of seeing that difference by chance alone is less than 5%. This 5% threshold is the alpha level. If you run 100 tests on identical variants with no true difference, random noise will produce a significant result in approximately five of them. That rate of false positives is the price of the 95% confidence standard. Raising the confidence level to 99% cuts false positives to 1% but requires a larger sample size to detect the same effect. The significance level you choose before the test determines the z-score threshold used in the sample size formula. A 90% confidence level uses a z-score of 1.645, 95% uses 1.960, and 99% uses 2.576. Higher thresholds demand more evidence, which translates directly into more required visitors.

How does baseline conversion rate impact sample size?

Baseline conversion rate has a nonlinear relationship with required sample size. Tests with insufficient sample size may show false positives or fail to detect real improvements. For a fixed relative MDE and significance level, variance is highest near 50% and lowest near 0% or 100%. A test detecting a 10% relative lift at 95% confidence and 80% power needs roughly 15,000 visitors per variant when baseline is 5%, around 38,000 per variant at 20%, and over 46,000 per variant at 50%. That difference exists because the binomial variance of a conversion rate p is p(1 - p), which peaks at p = 0.5. Low-conversion funnels (1% to 5%) have small absolute variance, so detecting even a modest relative change requires fewer observations than the same relative change on a high-conversion page. This relationship reverses when you fix absolute MDE rather than relative MDE. Detecting a 1 percentage point lift from a 2% baseline (a 50% relative improvement) is far easier than detecting the same 1 point lift from a 50% baseline (only a 2% relative improvement).

What happens if you run tests with too few visitors?

Running a test with too few visitors produces unreliable results in both directions. Underpowered tests have low statistical power, meaning they often fail to detect real improvements even when one exists. If your test has 50% power and a真实 improvement exists, you have a coin-flip chance of concluding there is no effect. That wasted effort and traffic could have been allocated to a better-powered test or a different optimization. The second risk is false positives. Even with a proper alpha level, peeking at results before reaching the planned sample inflates the effective false positive rate. If you check significance every day for two weeks, you give randomness 14 chances to cross the threshold. Researchers call this repeated testing problem p-hacking or optional stopping. The combination of low power and repeated testing creates a scenario where you miss real winners and ship false positives. Both errors cost revenue. The solution is to calculate required sample size up front, wait until both variants reach that count, then evaluate significance exactly once.

Once you identify a winning variant, use our conversion rate lift calculator to quantify the revenue impact of the improvement. For ad-driven tests, calculate your target metrics with the Google Ads break-even calculator to ensure your test goals align with profitability thresholds.