How the Calculator Works
The sample size tab uses the standard formula for comparing two conversion rates with a two-sided two-proportion z-test:
n per variant = [ zα/2 × √(2p̄(1 − p̄)) + zβ × √(p1(1 − p1) + p2(1 − p2)) ]² / (p2 − p1)²
- p1 is your baseline conversion rate for the exact page and goal you are testing.
- p2 is the baseline multiplied by (1 + minimum detectable effect).
- p̄ is the average of p1 and p2.
- zα/2 is 1.96 at 95% confidence, and zβ is 0.84 at 80% power.
With more than one variant, the calculator divides alpha by the number of comparisons against control (a Bonferroni correction), which keeps the overall false positive rate at the confidence level you chose. Our guide to A/B test sample size walks through the formula step by step with tables for common B2B conversion rates.
Worked Example
Illustrative numbers: a demo request page converts at 3% and gets 2,000 visitors a week. You want to know whether a new headline lifts conversion by at least 20%, so p2 is 3.6%. The calculator returns about 13,900 visitors per variant, 27,800 in total, which is 14 weeks of traffic. That is the honest answer for many B2B pages, and it is why the tool flags any test longer than a quarter.
Three ways to bring the duration down: test a bolder change and raise the minimum detectable effect to 30% or more, measure a more frequent event such as clicks to the demo page, or run the test on a higher-traffic page. If you are choosing between an A/B test and a test of several elements at once, read A/B testing vs multivariate testing first, because multivariate tests need far more traffic.
Reading a Significance Result
The significance tab compares two finished arms. Suppose the control had 5,000 visitors and 150 conversions (3.0%) and the variant 5,000 visitors and 190 conversions (3.8%). The relative lift is 26.7%, the p-value is about 0.027 and the 95% confidence interval of the difference runs from about 0.1 to 1.5 percentage points. The result is significant, but the interval is wide: the true lift could be small. Treat the measured lift as an upper-leaning estimate, especially when the test was stopped early.
For what to test first on a B2B site, see our B2B A/B testing strategy. If you test personalized experiences rather than single page changes, Markettailor's A/B testing runs holdouts per visitor segment.
Frequently Asked Questions
How many visitors do I need for an A/B test?
It depends on your baseline conversion rate and the smallest lift you want to detect. At a 3% baseline, detecting a 20% relative lift at 95% confidence and 80% power takes about 13,900 visitors per variant. Halving the lift you want to detect roughly quadruples the sample.
How is statistical significance calculated for an A/B test?
This calculator uses a two-sided two-proportion z-test. It pools both arms to estimate the noise you would expect if there were no real difference, divides the observed difference in conversion rates by that standard error, and converts the resulting z-score to a p-value. A p-value below 0.05 is significant at 95% confidence.
How long should an A/B test run?
Until it reaches the sample size calculated before launch, and for at least one full week so weekday and weekend behavior are both included. The duration estimate above divides the total sample by your weekly traffic to the tested page.
Can I stop an A/B test as soon as it is significant?
No. Checking results repeatedly and stopping at the first significant reading inflates the false positive rate well above 5%. Decide the sample size up front and read the result once it is reached, or use a method designed for continuous monitoring such as a sequential test.
What if my B2B site does not have enough traffic to A/B test?
Test bigger changes, choose a conversion event that happens more often (such as clicks to the demo page rather than booked demos), test on your highest-traffic template, or accept a longer test. If a test would take more than a quarter, use qualitative research and holdout comparisons instead.