What Is an A/B Test Sample Size and How Do You Calculate It?

What Is an A/B Test?

An A/B test compares a control version against one variation to see whether a single change produces a real difference in conversion rate. In practice, you split traffic between two versions of a page, email, ad, or flow. One version stays unchanged. The other changes one element you want to test, such as a headline, form length, CTA text, or pricing layout. At the end, you compare conversion rates between the two groups. For example, if version A converts at 5% and version B converts at 5.5%, the raw lift is 10% relative. The next question is whether that gap is large enough, and supported by enough visitors, to treat as a real result instead of random variation.

Why Sample Size Matters in A/B Testing

Sample size matters because an A/B test without enough visitors is more likely to give you a misleading result. Two problems show up when the sample is too small: - You can miss a real improvement because the test is underpowered. - You can stop on a fake winner because the early result looks stronger than it really is. This is why most sample-size tools ask for four inputs before the test starts: 1. Baseline conversion rate 2. Minimum detectable effect, or MDE 3. Significance level 4. Statistical power Those inputs define the trade-off. A smaller MDE means you want to catch a subtler lift, which pushes the required sample up. Higher confidence and higher power also increase the required visitor count. If your traffic is limited, the main decision is usually not "Can we test?" but "What size improvement is big enough to wait for?"

How to Calculate A/B Test Sample Size

You calculate A/B test sample size by combining your baseline conversion rate, target lift, significance level, and power in a two-proportion z-test power calculation. That sounds heavier than it is. The workflow is simple: 1. Start with your current conversion rate. 2. Decide the smallest lift worth acting on. 3. Set your false-positive threshold, usually 5%. 4. Set power, usually 80%. 5. Calculate the visitors needed per variant. The OnSumo A/B test sample size calculator uses that logic with the standard two-proportion z-test setup. The same method appears in power-analysis tools for two independent proportions, including statsmodels and NIST guidance for proportion tests. Here is a worked example that matches the tool's setup: - Baseline conversion rate: 5.0% - Minimum detectable effect: 10% relative lift - Expected variant conversion rate: 5.5% - Significance level: 95% two-sided - Power: 80% With those inputs, the required sample is about 31,237 visitors per variant, or 62,474 total. If each variant gets 1,000 visitors per day, the test needs roughly 32 days. The key pattern is the one most teams miss: lowering the MDE increases the sample fast. If you change the goal from detecting a 10% lift to detecting a 20% lift, the run time drops hard because the effect is easier to spot.

What Statistical Significance Means

Statistical significance means the observed gap is unlikely to be random chance at the threshold you chose before the test started. For a typical A/B test, a 95% significance level means you are accepting a 5% false-positive risk under the null hypothesis. That does not mean the variant is "95% likely to win forever." It means the observed difference cleared the threshold for this test design. Significance also does not tell you whether the result matters for the business. A tiny lift can be significant with enough traffic. A meaningful lift can fail to reach significance if the test did not collect enough data. That is why significance should be read with three other numbers: - The observed conversion rates - The absolute or relative lift - The planned sample size Once the test is complete, the next step is not to guess from the uplift alone. Check the finished result with the OnSumo A/B test significance calculator, then compare the business impact with tools such as the conversion rate lift calculator.

Common A/B Test Mistakes

The most common A/B test mistakes are setting no sample-size target, peeking early, and testing changes too small for the available traffic. Here are the mistakes that waste the most time: ### Choosing the wrong baseline Use recent conversion data from the page or flow you are actually testing. A site-wide average hides big differences between landing pages, checkout flows, and high-intent pages. ### Picking an unrealistic MDE If your traffic is low, asking the test to detect a 2% relative lift can make the required duration too long to be useful. Pick the smallest lift that would change a real decision. ### Stopping the moment the graph looks good Early spikes are common. If you stop before the planned sample is reached, you increase the odds of acting on noise. ### Mixing sample-size planning with result checking Pre-test planning asks, "How much traffic do we need?" Post-test analysis asks, "Did the measured gap clear the threshold?" Those are related steps, but they are not the same calculation. ### Ignoring duration and traffic split A sample-size number only helps if you compare it with actual daily visitors per variant. A test that needs 50,000 visitors per arm is not practical if each arm gets 300 visitors a day.

Frequently Asked Questions