Free calculator
A/B Test Significance Calculator
Enter the visitors and conversions for each variant. The calculator runs a two-proportion z-test and tells you whether the difference is statistically significant or still noise.
Verdict at 95% confidence
Not significant yet
A converted at 3.0%, B at 4.0% — a 33.3% lift.
Confidence level
94.1%
Two-tailed p-value 0.059. Below 95%, treat the difference as noise.
Runs in your browser. Nothing you enter is stored or sent.
How to read the result
The confidence level is the probability the difference you measured isn't luck. The usual bar is 95%: below it, keep the test running. The p-value is the same thing inverted — under 0.05 clears the bar.
The mistakes that invalidate tests
Checking daily and stopping the moment it crosses 95% inflates false positives — decide the sample size or duration up front. Run full weeks so weekday and weekend shoppers are both represented, and change one thing per test.
Full walkthrough: What is conversion rate optimization for a product page?.
More guides on this topic
Compare apps
Frequently asked questions
How many visitors do I need for an A/B test?
It depends on your conversion rate and the size of the lift you're trying to detect. As a rule of thumb, detecting a 20% relative lift on a 2% conversion rate takes several thousand visitors per variant — small stores often need weeks. If the calculator shows low confidence, the honest answer is to keep running.
What does statistically significant mean in an A/B test?
It means the difference between variants is large enough, given the sample size, that chance alone is an unlikely explanation. At 95% confidence, there's less than a 5% probability you'd see a gap this large if the variants actually performed the same.
Can I stop my A/B test as soon as it hits 95% confidence?
You shouldn't. Confidence fluctuates while data comes in, and stopping at the first crossing ('peeking') sharply raises the false-positive rate. Set a sample size or duration before you start, then read the result once at the end.
Why does my test say B wins but sales didn't improve?
Common causes: the test measured a shallow metric (clicks, not purchases), it was stopped early on a lucky streak, or the audience mix changed after launch. Significance is about the data you collected, not a guarantee the lift generalizes.