Here is the pattern I keep seeing: paid media teams are under pressure to ship creative refreshes on fixed dates, while experimentation teams are trying to run clean tests that can actually support a high-stakes decision. Both groups are doing sensible work. The conflict starts when they touch the same funnel at the same time and expect one test to answer every question.
In the Flextra thread, a landing page test was proposed, phone tracking was not ready, and campaign-level ad tests were already in motion. Marketing wanted learnings before creative rollout. CRO needed enough runtime, stable conditions, and complete measurement. That is the whole story.
what this problem actually is
This is not a fight about who understands statistics better. It is a systems problem. Two teams are optimizing for different decision types with different clocks.
Marketing is asking: Which campaign direction should we back right now? CRO is asking: Which change caused the lift, and can we trust it enough to scale? Those are different questions. If you do not separate them early, the test plan will drift into confusion.
clear up the technical confusion first
1) this is usually not an SRM issue
SRM is about traffic split imbalance. In your case, the bigger risks are selection effects, interaction between layers, and messy attribution.
2) ad tests on PCV can be valid
If the landing page is constant, an ad test on Product Chart Views can answer a business question cleanly: which ad brings more qualified sessions into that step of the funnel.
3) the trouble begins when ads and landing pages both change
When both layers move, you cannot cleanly isolate ad effect from page effect unless you designed for interaction from the beginning. Most teams have neither the traffic nor the setup for that every week.
how to think about it before launch
Use one forcing question at kickoff: Are we running a decision-grade test or a directional read?
If it is decision-grade, you protect test integrity even if that means a slower answer. If it is directional, you move faster and accept uncertainty on attribution. Most teams fail because they want the speed of directional work with the confidence language of decision-grade evidence.
a practical testing framework you can run this week
- State the decision in one sentence. Example: choose next quarter creative direction, or approve a permanent page change.
- Label the test type: decision-grade or directional.
- Freeze what stays constant. If you are testing ads, keep page experience fixed. If you are testing landing pages, hold ad inputs stable where possible.
- Confirm instrumentation before launch. If phone conversions matter and tracking is not ready, write that limitation into the decision memo.
- Set runtime and stop rules in advance. Do not retrofit rules when interim numbers look exciting.
- At readout, separate what you learned from what you proved. This single sentence prevents most stakeholder overreach.
three valid operating choices
option A: sequence ad test then page test
This is usually the cleanest path. It takes longer, but attribution stays readable and later decisions are safer.
option B: run both for directional signal
Use this when campaign timing is fixed. Be explicit that the result is directional and may blend effects. Do not present it as causal proof.
option C: factorial design
Best for isolating interaction, but traffic demand and planning load are high. Many teams should reserve this for major bets rather than every sprint.
FAQ
Is marketing wrong to ask for speed?
No. Speed can be rational. The mistake is asking for a fast directional read and then speaking about it as if it were decision-grade evidence.
Can we still optimize to PCV while we test?
Yes, if everyone understands what layer is being tested and what remains fixed. PCV itself is not the root problem.
What should we do when measurement is incomplete?
Either delay the decision-grade test until instrumentation is complete, or proceed with a clearly labeled directional read and documented blind spots.
What single question should we ask before every launch?
Ask: Is this a decision-grade test or a directional read? That one question resets expectations before the data arrives.
sources
Kohavi et al. rules of thumb for controlled experiments
Google Ads experiments documentation
NIST design of experiments handbook
American Statistical Association statement on p-values
next step
If you want this conflict to disappear, add a one-page test charter to your launch process and require teams to declare test type before anyone presses go.