A/B vs Multivariate vs Multi-Page Testing: Which Should You Run?

The Test Type Decision Has Real Consequences

Most teams default to A/B tests for everything. A smaller number have heard that multivariate testing is "more sophisticated" and try to run MVTs on pages that get 20,000 visitors a month. Both approaches lead to wasted time.

Choosing the wrong test type doesn't just slow you down — it can produce statistically meaningless results, cause your program to stall waiting for significance that will never arrive, or give you interaction data you can't interpret because you don't have the traffic to support it.

Here's the decision framework, with the actual numbers behind each choice.

Quick Decision Tree

Before diving into each type, here's how I make the call:

Are you testing one distinct idea (a redesign, a new headline, a different CTA)? Use A/B.

Do you have 100,000+ monthly visitors to the tested page and want to understand how multiple specific elements interact? Consider MVT.

Does your test require users to see a consistent variation across multiple pages (e.g., checkout flow)? Use multi-page.

Are you validating your platform setup or baselining before real tests? Use A/A (see our guide on How to Run an A/A Test in Optimizely).

If you answered A/B to everything, you're right about 80% of the time. Here's why.

A/B Testing: The Default for a Reason

An A/B test in Optimizely splits traffic between a control and one or more variants. Each user sees one experience. You measure which experience produces better outcomes on your primary metric.

Traffic requirements: Manageable. If your page converts at 3% and you want to detect a 15% relative lift, you need roughly 17,000 visitors per variation. At 50,000 monthly visitors to the page, that's achievable in about two weeks with a 50/50 split.

When it works best:

• Testing one major design direction against another
• Single-element changes where you're confident about which element matters (headline, hero image, CTA placement)
• Pages with moderate traffic (10,000-100,000 monthly visitors)
• Early-stage programs building testing culture and velocity

When it breaks down:

• You want to know whether Headline A works better with Image A or Image B — A/B can't answer interaction questions
• You have so many ideas you want to test simultaneously that sequential A/B tests would take years

**Pro Tip:** Running an A/B test with 3-4 variants is still an A/B test — Optimizely handles multi-arm experiments natively. But each additional variant splits your available traffic further, extending time to significance. Four variants at 25% each take roughly twice as long as two variants at 50% each to reach the same statistical threshold.

Multivariate Testing: Powerful, But Frequently Misused

An MVT in Optimizely tests multiple page elements simultaneously by creating all possible combinations of your variations. If you have a headline section (2 options) and a hero image section (2 options) and a CTA section (2 options), Optimizely creates 8 combinations and routes traffic to all of them.

The 64-combination hard limit. Optimizely caps MVT at 64 combinations. That sounds like a lot until you realize: 3 sections × 3 variations each = 27 combinations. 4 sections × 3 variations each = 81 combinations — you've already hit the cap.

The traffic math that kills most MVTs. Traffic is split across all combinations. Here's the formula:

Visitors per combination per week = (Page visitors/week × Traffic allocation) / Number of combinations

Worked example: Your homepage gets 50,000 visitors per month (~12,500/week). You run a 100% traffic allocation MVT with 8 combinations (2 headlines × 2 images × 2 CTAs).

Each combination gets: 12,500 / 8 = 1,562 visitors per week.

Your baseline homepage conversion rate is 2.5%. At 1,562 visitors/week per combination, you need approximately 4,000 visitors per combination to detect a 20% relative lift (from 2.5% to 3.0%). That's 2.5 weeks per combination — which sounds fine, but all 8 combinations run simultaneously, so you need 4,000 per combination total, not per week. With 1,562/week, you're looking at ~2.5 weeks total. That's actually feasible.

Now scale it up: 6 sections × 2 variations = 64 combinations. Each combination gets 12,500 / 64 = 195 visitors per week. To reach 4,000 visitors per combination, you need 20 weeks. That's a 5-month test on a 50,000-visitor-per-month page. For most testing programs, this is a practical dead end.

The rule of thumb: MVT requires roughly 1,000 visitors per combination per week to be tractable. Count your combinations before you launch.

**Pro Tip:** Use Optimizely's "section rollup" feature for MVTs with many combinations. Section rollup collapses the data for one section across all other sections' variations, effectively giving you an A/B-style analysis of that one element with the combined traffic of all combinations. It's the escape hatch when your MVT runs long.

When MVT is actually the right call:

• You have a very high-traffic page (500,000+ monthly visitors to that page)
• You genuinely need to know how elements interact (does the benefit-focused headline only work with the product image, not the lifestyle image?)
• You've already run sequential A/B tests on the individual elements and found conflicting results that suggest interaction effects

When MVT is a trap:

• You have under 100,000 monthly visitors to the tested page
• You're using MVT because you have "a lot of ideas" and want to test them all at once (run them sequentially instead)
• You haven't validated your individual hypotheses yet — MVT is for optimization, not exploration

Full Factorial vs. Partial Factorial

Optimizely offers two MVT traffic allocation approaches:

Full factorial tests every combination against every other combination. Traffic is allocated at the section level — each section's variations receive their allocated percentages, and Optimizely routes users to combinations probabilistically. The math is clean and statistically rigorous. The downside: you may end up testing combinations that are logically incoherent (e.g., a blue button on a blue background if color is one of your sections).

Partial factorial allocates traffic at the combination level, which lets you manually zero out specific combinations. This is useful when some combinations don't make design sense. The tradeoff is reduced statistical power for the interactions you do test, and the analysis becomes more complex.

For most practitioners, start with full factorial and use section rollup for interpretation. Use partial factorial only if you have specific combinations that would genuinely break the user experience.

**Pro Tip:** If you find yourself needing to exclude more than a quarter of your combinations in partial factorial, your test design is too complex. Simplify it or break it into sequential A/B tests.

Multi-Page (Funnel) Testing

Multi-page tests ensure that users experience a consistent variant across multiple pages throughout a session. The classic use case: checkout funnel testing.

The problem multi-page tests solve: If you run separate A/B tests on your product page and your cart page simultaneously, a user who saw Variant A on the product page might see Variant B on the cart page (or Control). Now you're measuring the performance of mixed experiences, not pure variants. Your results are confounded.

Multi-page tests in Optimizely bucket users once and follow them with a consistent experience across all tested pages.

Traffic requirements: Similar to A/B tests, but be aware that your primary metric should typically be measured at the end of the funnel. If you're running a multi-page test across product page → cart → checkout, the meaningful metric is checkout completion — not add-to-cart rate. Checkout completion rates are typically 1-3%, which means you need substantially more traffic than a top-of-funnel metric would require.

Practical example: An e-commerce client was testing a "simplified checkout" concept that required changes to three pages: the cart, the address form, and the payment form. Running three separate A/B tests would mean users could experience the simplified cart with the old address form — a combination that doesn't represent either hypothesis. Multi-page test solved this cleanly.

Constraints to know:

• All pages in a multi-page test must share targeting conditions. If your product page targets all visitors but your checkout page is only accessible by logged-in users, you can't cleanly include both.
• All pages must have the same number of variations. You can't have two variants on page 1 and three on page 2.

**Pro Tip:** For checkout flow tests, always include a macro conversion metric (order completed) even if you're primarily interested in a micro metric (checkout page engagement). Checkout optimization that increases page engagement but reduces order completions is a very common failure mode.

The "Start A/B, Graduate to MVT" Principle

The right progression for a testing program:

1. A/B test individual elements to understand the directional impact of each (does a short form outperform a long form? does a testimonial near the CTA help?).
2. Once you have directional answers on individual elements, if your traffic supports it and you suspect interactions, run an MVT to understand how the winning elements combine.
3. Use multi-page tests whenever your hypothesis spans multiple steps of a user flow.

Most programs never need to graduate beyond step 1 for at least the first year. Velocity of A/B tests almost always generates more learning than depth of MVTs.

Common Mistakes

Running MVT on low-traffic pages to "save time." MVT on a 30,000 visitor/month page with 8 combinations will take 3-4 months to reach significance. Six sequential A/B tests, each running 2 weeks, cover the same ground in the same time with far clearer results.

Using multi-page tests when A/B would work. If your change only affects one page, use A/B. Multi-page tests add complexity without benefit unless you genuinely need cross-page consistency.

Confusing "testing multiple variations" with "multivariate testing." An A/B test with four variants (A/B/C/D) is not an MVT. MVT specifically tests combinations of element-level variations.

Launching an MVT without checking the combination count. Always verify total combinations before launch. Optimizely will let you build a 48-combination MVT. It won't tell you that your traffic supports running a maximum of 12.

What to Do Next

1. For your next experiment, apply the decision tree at the top of this article before touching Optimizely. Write down which test type you're choosing and why.
2. If you're considering MVT, calculate your estimated visitors per combination per week using the formula above. If it's under 500, run sequential A/B tests instead.
3. Review our guide on How Long Should You Run an A/B Test? — the duration math changes significantly based on which test type you're running.
4. If you're optimizing a checkout flow, map out every page in the flow before you build your Optimizely experiment. Identify which pages need consistent variant exposure and set up a multi-page test from the start.