The difference in A/B testing vs multivariate testing is what each one compares. An A/B test compares two versions of a page. An A/B/n test compares three or more complete versions. A multivariate test (MVT) changes several elements at once, such as headline, hero image and CTA, and tests every combination (sometimes called multivariate A/B testing) to learn which elements and which combination drive conversions.
For B2B websites the choice is mostly decided by traffic. Each extra variant or combination divides your visitors further, and the sample size math grows faster than most teams expect. This post compares the four test types B2B marketers actually use (A/B, A/B/n, multivariate, and segment-targeted personalization tests), computes what each one costs in traffic, and gives a decision table plus examples of good tests for each.
The Four Test Types at a Glance
| Test type | What it compares | Question it answers | Traffic cost |
|---|---|---|---|
| A/B | Control vs one variant | Is this version better than what we have? | Lowest |
| A/B/n | Control vs 2+ complete variants | Which of these distinct approaches is best? | Grows with each arm, plus a penalty for multiple comparisons |
| Multivariate (MVT) | Every combination of several elements | Which elements matter, and do they interact? | Highest, especially for interactions |
| Segment-targeted (personalization) test | Personalized experience vs holdout, within one segment | Does tailoring this page to this segment work? | Depends on segment traffic and holdout size |
The Traffic Math Behind A/B Testing vs Multivariate Testing
All numbers below use the standard two-proportion sample size formula at 95% confidence (two-sided) and 80% power, computed in Node.js. The method and full tables are in our post on A/B test sample size. You can run the same numbers for your own page with our A/B test sample size calculator, which applies the Bonferroni correction for A/B/n tests.
Example (illustrative numbers): a demo page converts at 3%, and you want to detect a 20% relative lift (3.0% to 3.6%). A simple A/B test needs 13,914 visitors per variant, or 27,828 in total.
A/B/n: more arms, stricter thresholds
With three challengers against one control you make three comparisons, and the chance of at least one false winner rises. The simplest correction (Bonferroni) divides alpha by the number of comparisons, so each comparison is tested at 0.05 / 3 = 0.0167. That raises the per-arm sample from 13,914 to 18,559, and the total from 27,828 to 74,236 across four arms. Roughly 2.7x the traffic of an A/B test.
Multivariate: combinations multiply
Now test three elements with two versions each (headline A or B, image A or B, CTA A or B). A full factorial design has 2 × 2 × 2 = 8 cells. If you want to compare each combination with the control, as most MVT reports implicitly do, each cell needs a full sample: 8 × 13,914 = 111,312 visitors without any correction, or 8 × 22,112 = 176,896 visitors with a Bonferroni correction for seven comparisons.
Here is the part most comparisons leave out. A factorial design is actually efficient for main effects. To estimate the headline effect, you compare the four cells with headline A against the four with headline B, which uses all your traffic. If the elements do not interact, 27,828 visitors tell you about all three elements at once. But then you have run three A/B tests in parallel, which is useful, though it is not what people buy MVT for.
The reason teams run MVT is to find interactions ("this headline only works with that image") and the single best combination. Interactions are expensive. Andrew Gelman's widely cited note, You need 16 times the sample size to estimate an interaction than to estimate a main effect, shows why: an interaction estimate has twice the standard error of a main effect, and if the interaction is half the size of the main effect (his stated assumption), you need 16 times the sample for the same power. Applied to our example, that is about 445,000 visitors.
What that means in months
| Design (3% baseline, 20% MDE) | Total visitors | Months at 5,000/month | Months at 20,000/month |
|---|---|---|---|
| A/B | 27,828 | 5.6 | 1.4 |
| A/B/n with 4 arms (Bonferroni) | 74,236 | 14.8 | 3.7 |
| MVT 2×2×2, each cell vs control (uncorrected) | 111,312 | 22.3 | 5.6 |
| MVT 2×2×2, each cell vs control (Bonferroni) | 176,896 | 35.4 | 8.8 |
| MVT interaction, half-size (16x rule) | 445,248 | 89.0 | 22.3 |
On a page with 20,000 monthly visitors, a 2×2×2 MVT that reads each combination takes most of a year. On 5,000 it takes years. That is why we think multivariate testing is rarely viable for B2B websites, and why "we ran an MVT and combination 6 won" on B2B traffic should be treated with suspicion.
What Most Teams Get Wrong
The common mistake is choosing the test type by ambition rather than by question. Teams pick MVT because they have many ideas, not because they need to know whether the ideas interact. They pick A/B/n with five variants because nobody could agree on one. Both choices split limited traffic into slices too thin to read.
A second mistake is testing elements when the real question is about audiences. On a B2B site, the biggest difference in what converts is often who is looking, not which headline is shown. A manufacturing buyer and a SaaS founder want different proof points. Testing headline A vs B for everyone averages those audiences together and can hide a variant that wins strongly for one segment and loses for another. That is the case for segment-targeted personalization tests.
Decision Table: Which Test to Run
| Situation | Monthly visitors to the page | Elements changed | Recommended test |
|---|---|---|---|
| One clear hypothesis, big change | Any, including under 5,000 | One concept (may touch several elements) | A/B |
| Two or three genuinely different approaches | 15,000+ | Whole-page concepts | A/B/n with 3 arms max |
| Need to know if elements interact | 100,000+ | 2 to 3 elements, 2 versions each | Multivariate (full factorial) |
| Different audiences likely want different things | Enough traffic within the segment | Content tailored to the segment | Segment-targeted test vs holdout |
| Not enough traffic for any of the above | Under 2,000 | Any | Qualitative research, then an A/B test on a bigger change |
The traffic thresholds are rough guides for a 3% baseline. Compute your own with the sample size formula before committing.
Examples of Good Tests for Each Type
A/B tests
- Demo form vs instant calendar booking on the demo page.
- Feature-led hero vs outcome-led hero on the homepage.
- Pricing page with a "talk to sales" tier vs published prices for every tier.
Bundling several elements into one concept is fine in an A/B test. You learn whether the concept wins, not which element did it, and that is usually the decision you actually need to make. Our B2B A/B testing strategy covers how to prioritize these ideas.
A/B/n tests
- Three value propositions on a high-traffic paid landing page: speed, cost savings, risk reduction.
- Three CTA offers: demo, ROI assessment, product tour. See our guide to data-driven personalized CTAs for how offer type changes by stage.
Keep A/B/n tests to three arms (control plus two) unless traffic is unusually high, and make each arm meaningfully different so the effects are large enough to separate.
Multivariate tests
- Headline (2 versions) × social proof type (logos vs a case study quote) on a homepage with very high traffic, when you have a specific reason to expect the two to interact.
If you cannot name the interaction you expect, you probably do not need an MVT.
Segment-targeted personalization tests
- Industry-specific homepage hero for visitors from financial services companies vs the default hero.
- Enterprise-focused proof (security, scale) on the pricing page for companies above 1,000 employees vs the standard page.
- Target-account greeting and case study for ABM accounts vs the generic page.
How to Test Personalization Against a Holdout
A personalization test is an A/B test run only inside a segment. The control is a holdout group of that segment who see the default page. Without the holdout, you cannot separate the effect of personalization from the fact that the segment may simply convert differently.
- Define the segment precisely, for example "visitors identified as companies in financial services with 200+ employees." Only those visitors enter the test.
- Measure segment traffic to the tested pages per month. This, not total site traffic, sets your duration.
- Choose the holdout size. A 50/50 split is the fastest to read. Smaller holdouts protect more visitors from the default experience but cost time.
- Pick a conversion that the segment actually does, measured over the visit or the account's visits, and keep pipeline as a guardrail metric.
- Fix the sample size and duration before launch, and read the result only for that segment.
Example (illustrative numbers): a segment brings 3,000 visitors a month, converts at 3%, and you want to detect a 30% lift. Computed with an unequal-allocation version of the formula:
| Holdout share | Total visitors needed | Months at 3,000/month |
|---|---|---|
| 50% | 12,903 | 4.3 |
| 20% | 18,639 | 6.2 |
| 10% | 32,234 | 10.7 |
A 10% holdout takes about 2.5 times longer to read than a 50/50 split. Our recommendation: run the first test for each segment at 50/50 to get a clean answer, then keep a small permanent holdout (10 to 20%) after rollout to track whether the effect holds over time. The personalization testing roadmap shows how to sequence these tests across segments, and the State of B2B Personalization 2026 report is good background on which segments teams typically prioritize.
Running these holdout tests requires identifying the segment before the page renders and splitting only that segment. Markettailor's A/B testing does this per segment, using company-level segmentation to decide who enters each test, so a fintech test only ever sees fintech visitors.
Frequently Asked Questions
What is the difference between A/B testing and multivariate testing?
A/B testing compares two complete versions of a page. Multivariate testing changes several elements at once and tests all combinations, so it can show which elements matter and whether they interact, at the cost of far more traffic.
What is A/B/n testing?
A/B/n testing compares a control with two or more complete variants at the same time. It is useful when you have a few genuinely different approaches, but each extra arm needs its own sample and a stricter significance threshold.
How much traffic do you need for multivariate testing?
For a 2×2×2 test on a page converting at 3%, reading each of the 8 combinations against control for a 20% lift needs roughly 111,000 to 177,000 visitors, depending on the correction for multiple comparisons. Detecting interactions can take several times more.
Is multivariate testing worth it for B2B websites?
Rarely. Most B2B pages lack the traffic to read combinations or interactions within a few months. A/B tests of bigger concepts and segment-targeted personalization tests usually produce clearer answers faster.
Can you combine A/B and multivariate testing?
Yes. A common sequence is to A/B test concepts first, then, on very high-traffic pages only, run a small factorial test on two elements within the winning concept if you have a specific interaction to check.