The Misconception That's Wasting Your Testing Capacity
A lot of teams treat A/B testing and multivariate testing (MVT) as two ways to accomplish the same goal — with MVT being the "more advanced" option for when you want to test multiple things at once. This framing is wrong, and it leads teams to either run MVT when they don't have the traffic to support it, or run A/B tests sequentially when the interaction between elements is actually what they need to understand.
They're different tools that answer different questions. Using the wrong one doesn't just waste time — it gives you answers to questions you didn't ask.
What Each Test Type Actually Tests
A/B testing isolates one variable and answers one question: does this specific change improve this specific metric, when everything else remains constant?
The power of A/B testing is the isolation. You change the headline, nothing else changes. If conversion improves, the headline was the cause. The causal interpretation is clean.
The limitation: you're testing one element in a sea of other elements you're not testing. The new headline might perform well in combination with the current button copy, but if you later test a new button copy, you might find the combination of new headline + new button copy performs worse than either alone. You won't know this unless you test them together.
Multivariate testing tests multiple elements simultaneously across all their combinations. If you're testing element A (2 variants) and element B (2 variants), you're testing 4 combinations: A1+B1, A1+B2, A2+B1, A2+B2. The question MVT answers: which combination of changes produces the best outcome? And — crucially — do the elements interact?
The Interaction Effect: What MVT Gives You That A/B Can't
The interaction effect is the insight that justifies MVT. It occurs when the effect of one element depends on the state of another element.
Example: You're testing headline copy (feature-led vs. outcome-led) and hero image (product screenshot vs. lifestyle photo). In isolation:
- Outcome-led headline wins by +6% over feature-led
- Lifestyle photo wins by +4% over product screenshot
But in combination, the lifestyle photo + feature-led headline outperforms all other combinations by 15%. The headline and image have an interaction effect: the lifestyle photo works better with feature-led copy because the photo already communicates the outcome (the lifestyle), and the feature copy then grounds it in the specific mechanism. The outcome-led headline + lifestyle photo is redundant and loses.
Sequential A/B testing would have told you: "use outcome-led headlines and lifestyle photos." MVT tells you: "use feature-led headlines with lifestyle photos." These are opposite conclusions, and only MVT can surface the interaction.
The practical question: how often do interaction effects occur in practice? In my experience, they're real but not universal. For independent elements (a headline and a footer change), interactions are rare. For co-dependent elements (headline and hero image; CTA copy and CTA color; pricing layout and the featured tier), interactions are common and often meaningful.
**Pro Tip:** Before running an MVT, ask: are these elements visually or conceptually co-dependent? If a user's perception of one element changes based on the state of another, you have a candidate for interaction effects. If the elements are on different parts of the page and serve unrelated functions, sequential A/B testing is probably fine.
The Traffic Requirement Problem
Here's where MVT goes wrong for most teams. The traffic math is punishing.
For a standard A/B test, you need roughly N visitors per variation (where N is determined by your baseline CVR, MDE, and desired power). Two variations: 2N total visitors.
For MVT with k combinations, you need approximately N visitors per combination: k × N total visitors. This scales fast.
Worked Example
Suppose your page converts at 3%, you want to detect a 10% relative lift (0.3 percentage points), at 95% confidence and 80% power. Required sample per variation: approximately 48,000 visitors.
- A/B test (2 variations): 96,000 total visitors. At 20,000 visitors/week: ~5 weeks.
- MVT (2 elements × 2 variants = 4 combinations): 192,000 total visitors. At 20,000 visitors/week: ~10 weeks.
- MVT (3 elements × 2 variants = 8 combinations): 384,000 total visitors. At 20,000 visitors/week: ~19 weeks.
That 8-combination MVT takes nearly 5 months at 20K visitors/week. And 20,000 visitors/week to a single page is not a low-traffic site. If you have 5,000 visitors/week, that 8-combination MVT takes over a year.
Teams chronically underestimate how much traffic MVT requires. The result: they run underpowered MVTs, get inconclusive results, and either call a winner prematurely or waste months and learn nothing.
**Pro Tip:** Before scoping any MVT, calculate how long it will take given your page's actual traffic. If it exceeds 8 weeks, either reduce the number of combinations (cut an element or reduce variants per element), accept a larger MDE, or switch to sequential A/B testing.
The Rule of Thumb
You need roughly 100,000 page visitors per month to the specific page being tested to run a meaningful 4-combination MVT (2 elements × 2 variants each) with a 10% relative MDE at standard confidence. That's the floor.
For 8 combinations (3 elements), double it: ~200,000 page visitors/month.
Very few pages outside of homepages on mid-to-large sites will meet these thresholds. Enterprise e-commerce, major SaaS homepages, and high-traffic content pages can. Most product pages, landing pages, and blog CTAs cannot.
When A/B Testing Is the Wrong Choice
There are two situations where sequential A/B testing underperforms relative to MVT:
Situation 1: You're testing highly co-dependent elements. If you're redesigning a pricing page and you want to test: tier layout (cards vs. table), featured tier (middle vs. top), and CTA style (button vs. link), these three elements interact heavily. Running them as three sequential A/B tests will give you locally optimal results for each element, but may miss the globally optimal combination. If you have the traffic, MVT is the right tool.
Situation 2: You need to maximize learning per unit of time, and traffic is high. At very high traffic (500K+ monthly visitors to the test page), an 8-combination MVT runs in 3-4 weeks and gives you results that would take 6 sequential A/B tests (months) to replicate. If testing velocity matters and traffic allows, MVT compounds learning faster.
When MVT Is the Wrong Choice
This is the more common mistake. MVT is the wrong choice when:
Traffic is below ~100K monthly page visitors. The math simply doesn't work. You'll run an underpowered test and either get inconclusive results or prematurely call a winner.
The elements are independent. If there's no reason to expect interaction effects (e.g., testing a headline and a footer testimonial section on separate parts of a long page), sequential A/B tests are more efficient. Run the higher-impact element first.
You don't have the patience for a long test. If your business needs insights in 2-3 weeks, a 4+ combination MVT at moderate traffic will never deliver a valid result in that window. Choose between running an A/B test on the highest-impact element or accepting that you won't have a clean answer quickly.
You're still learning what elements matter. If you're early in your testing program and don't yet know which elements on a page have the highest impact, start with A/B tests. MVT is most valuable when you already know which elements matter enough to test together.
**Pro Tip:** If you're using Optimizely and want to run an MVT, use the built-in traffic calculator to project test duration before building. This is the most common place teams discover their page doesn't have enough traffic, and it's better to learn this before you've built 8 variants.
The Third Option: Multi-Page (Funnel) Testing
Most teams forget about this entirely. Multi-page testing — also called funnel testing or experiment chaining — lets you test changes across sequential pages in a funnel rather than multiple elements on a single page.
Use case: you're testing the checkout funnel. Page 1 is the cart. Page 2 is the payment details form. Page 3 is the order confirmation. You want to know whether changes to cart messaging affect downstream completion, not just cart-page metrics.
In Optimizely, this is done by creating a multi-page experiment that spans multiple URLs with sequential conditions. You can measure the impact of a change on Page 1 on metrics that occur on Pages 2 and 3.
This is the right tool when:
- You're testing anything in a multi-step funnel (checkout, sign-up, onboarding)
- The relevant conversion metric happens on a different page from the change
- You want to measure downstream effects, not just immediate click-through
The traffic requirement for multi-page testing is similar to A/B testing (you're still comparing two variants), but the metric you care about — funnel completion rate — is the denominator at the first page, not the last.
Decision Framework
| Test Goal | Traffic Level | Use | |---|---|---| | Test one element, clear causal interpretation | Any | A/B test | | Test co-dependent elements with potential interaction | >100K page visits/month | MVT | | Maximize learning velocity on a high-traffic page | >250K page visits/month | MVT (3-4 combinations) | | Measure cross-page funnel impact | Any | Multi-page experiment | | Test multiple independent elements on low traffic | Any | Sequential A/B tests | | Test pricing or high-stakes checkout changes | Any | A/B test (cleaner causal chain) |
How to Set Up Each Type in Optimizely
A/B test: Standard experiment setup. Set your traffic allocation (typically 50/50 for two variations), define your primary metric, and set your experiment duration based on your sample size calculation. Stats Engine handles sequential analysis automatically.
MVT: In Optimizely, create a Multivariate Experiment and define each "section" (page element) with its variants. Optimizely automatically generates all combinations. Important: check the estimated traffic required in the experiment setup before launching — Optimizely will warn you if the test duration is prohibitively long given your page's traffic.
Multi-page experiment: Create an experiment with "page" conditions that span multiple URLs in sequence. Set your primary metric to a conversion on the final page. Make sure your traffic allocation is set at the entry point of the funnel, not midway through.
Common Mistakes
Mistake 1: Running MVT without checking traffic requirements. The most common MVT failure. Build your variants, launch the test, check back in 8 weeks, find the confidence interval is too wide to conclude anything.
Mistake 2: Treating MVT results as a ranking problem. MVT doesn't tell you "element A matters more than element B." It tells you which combination performed best. The ranking of elements requires a different analysis (contribution analysis), and most standard MVT reports don't surface this cleanly.
Mistake 3: Running sequential A/B tests on co-dependent elements and missing interaction effects. If your sequential tests give contradictory results (new headline wins in test 1 but "hurts" in test 2 because the combined effect with the new image is negative), you have an interaction effect you're not measuring.
Mistake 4: Using MVT for a first test on a new page. You don't yet know which elements matter most. Start with A/B tests on individual high-impact hypotheses to build context, then use MVT to optimize the combination of the elements you've confirmed matter.
What to Do Next
- Identify the next test on your roadmap. Ask: are there two or more co-dependent elements I want to test simultaneously? If yes, check whether your page has the traffic to support MVT. Calculate required sample × combinations and divide by weekly traffic.
- If traffic supports MVT: define your sections, build your variants, and check Optimizely's estimated duration before launching.
- If traffic doesn't support MVT: prioritize your elements by estimated impact (heatmap data, user research, funnel analysis), and run sequential A/B tests starting with the highest-impact element.
- If you're testing a funnel: consider a multi-page experiment. Define the entry point, the change on the first page, and the downstream conversion metric you care about.
For a step-by-step guide to setting up A/B, MVT, and multi-page experiments in Optimizely — including traffic calculator walkthrough and Stats Engine configuration — see the Optimizely Practitioner Toolkit at atticusli.com/guides/optimizely-practitioner-toolkit.