A/B Significance: Winner or Just Noise?
A higher rate is not automatically a real win. A significance check asks whether the gap is bigger than random variation would usually produce.
What is the A/B Significance Checker?
Enter visitors and conversions for variants A and B, and the tool compares rates with a two-proportion z-test approximation at the 95% level. This is a simplified statistical check, not a full experiment analysis, so use it as a guide and confirm important decisions with proper analysis.
How to use it
- Enter visitors and conversions for variant A.
- Enter visitors and conversions for variant B.
- Click Check Significance.
Worked example
With 500 versus 580 conversions from 5000 visitors each, rates are 10% and 11.6%, and the z score is about 2.58, significant at 95%.
Does significant mean the change caused the lift?
Only if the test was run fairly, with random split traffic and no peeking or mid-test changes that bias the result.
Frequently asked questions
What does 95% mean here?
Roughly, a gap this big would appear by chance less than 1 time in 20 if there were no real difference.
Can I stop the test early?
Stopping the moment significance appears inflates false wins. Decide the sample size in advance and run to it.