A/B test statistical significance calculator
Is your split test result real, or just noise? Enter the visitors and conversions for each variation and find out whether you can trust the winner.
| Visitors | Conversions | Conversion rate | |
|---|---|---|---|
| Variation A | 18.50% | ||
| Variation B | 21.50% |
What statistical significance means for your test
Statistical significance tells you how likely it is that the difference between two conversion rates is real rather than the result of chance. This calculator runs a two-proportion z-test on the visitors and conversions you enter.
- 95% confidence (p below 0.05) is the usual threshold for calling a winner.
- 99% confidence (p below 0.01) is safer for decisions that are expensive to reverse.
- Below 95%, keep the test running. More visitors narrow the uncertainty.
Two practical rules: decide the sample size before you start, and do not stop a test the moment it reaches significance, because results fluctuate early on. A full week of traffic also smooths out weekday and weekend behaviour.
Split testing questions, answered
How many visitors do I need for an A/B test?
It depends on your current conversion rate and the uplift you hope to detect. As a rough guide, detecting a 20% relative improvement on a 5% conversion rate needs around 6,000 visitors per variation. Smaller uplifts need far more.
What does the p-value mean?
The probability of seeing a difference at least this large if the two variations actually performed the same. A p-value of 0.03 means that, if there were no real difference, a gap this big would appear about 3% of the time.
Can I test pop-ups with Convertful?
Yes. Convertful splits traffic between widget variations automatically and reports conversions for each one in live analytics, so you can paste the numbers here or read the winner in the dashboard.
Run A/B tests on your pop-ups without the spreadsheet
Convertful splits traffic, counts conversions and shows the winner for every widget. Free for 14 days.