How to Handle Multiple Metrics in A/B Testing

The Multiple Metrics Problem

Every meaningful A/B test tracks more than one metric. You have your primary metric, several secondary metrics, guardrail metrics, and probably a handful of diagnostic metrics that help you understand what happened.

This creates a problem that most teams handle badly. When metrics disagree — the primary goes up, a secondary goes down, a guardrail is flat — the decision about whether to ship becomes surprisingly complex.

Teams that handle multiple metrics well make better decisions and avoid the hidden costs of metric myopia. Teams that handle them poorly either cherry-pick the metrics that support their preferred outcome or paralyzed by conflicting signals.

Why Multiple Metrics Are Necessary

Before discussing how to handle multiple metrics, it is worth understanding why a single metric is insufficient.

Business Outcomes Are Multidimensional

No single metric captures the full picture of business health. Revenue is important, but revenue growth at the expense of customer satisfaction is unsustainable. Conversion rate matters, but conversion rate without retention is a leaky bucket.

Experiments that optimize a single metric in isolation can cause damage that only becomes visible months later. Multiple metrics provide the peripheral vision that prevents this.

Users Are Not Monolithic

A change might benefit one user segment while harming another. The aggregate metric might look flat — the gains and losses cancel out — but the user experience has meaningfully changed for both groups. Secondary metrics that track segment-specific behavior reveal dynamics that the aggregate hides.

Mechanisms Matter

Understanding why a change worked is as important as knowing that it worked. Multiple metrics illuminate the mechanism. If a new onboarding flow increases trial-to-paid conversion, secondary metrics tell you whether it is because users understand the product better (feature adoption up), engage more deeply (session frequency up), or simply felt more urgency (usage pattern unchanged, but conversion happened faster).

The mechanism determines whether the improvement is sustainable and how to build on it.

The Statistical Challenge

Tracking multiple metrics introduces statistical complications that teams routinely ignore.

The Multiple Comparisons Problem

If you test one metric at a ninety-five percent confidence level, you have a five percent chance of a false positive. If you test twenty metrics, the probability of at least one false positive is dramatically higher.

This is not a theoretical concern. Teams that track many metrics and declare success when any of them shows significance are fooling themselves. They are running a lottery, not an experiment.

Solutions to Multiple Comparisons

Pre-specify the primary metric. The most important solution is the simplest: decide which metric determines the ship decision before the test starts. Other metrics are informational, not decisional.

Apply corrections for secondary metrics. If you want to make statistical claims about secondary metrics, apply a correction like Bonferroni (divide your significance threshold by the number of metrics) or Benjamini-Hochberg (which controls the false discovery rate). These corrections are conservative, but they prevent false claims.

Use a hierarchical approach. Define an ordered list of metrics. Test the first metric at full significance. If it is significant, test the second at full significance. If it is not, stop. This fixed-sequence approach controls the family-wise error rate without being as conservative as Bonferroni.

Frameworks for Multi-Metric Decision-Making

The Decision Matrix

Create a decision matrix that maps metric outcomes to actions:

| Primary Metric | Guardrails | Secondary Metrics | Decision | |---|---|---|---| | Significant improvement | All pass | Mostly positive | Ship | | Significant improvement | All pass | Mixed signals | Ship with monitoring | | Significant improvement | One or more violated | Any | Do not ship — investigate | | No significant change | All pass | Some improvements | Do not ship — iterate | | Significant degradation | Any | Any | Kill the experiment |

Define this matrix before the test launches. When results come in, the decision is mechanical — you follow the matrix.

The Metric Hierarchy

Organize your metrics into tiers:

Tier 1: Decision metric (one metric). This is the primary metric. It must improve for the experiment to be considered a success.

Tier 2: Veto metrics (two to four metrics). These are guardrails. If any of them degrade beyond the predefined threshold, the experiment fails regardless of the primary metric.

Tier 3: Observation metrics (unlimited). These provide context and learning but do not influence the ship decision. You observe them, you learn from them, but you do not use them to override the decision made by Tiers 1 and 2.

This hierarchy prevents the common failure mode of promoting a secondary metric to decision status after the fact because it showed a positive result.

The Weighted Scorecard

For experiments where multiple outcomes are genuinely important and difficult to rank, use a weighted scorecard:

1. List all relevant metrics
2. Assign a weight to each based on business importance (weights must sum to one)
3. For each metric, calculate the effect size (relative change from control)
4. Multiply each effect size by its weight
5. Sum the weighted effects for an overall experiment score

A positive overall score means the experiment is net positive. A negative score means it is net negative.

The danger of this approach is that it can mask a severe degradation in one metric behind improvements in others. Always combine the scorecard with guardrail thresholds — no individual metric should degrade beyond an acceptable limit, regardless of the overall score.

Handling Common Metric Conflicts

Short-Term Metric Up, Long-Term Metric Down

This is the most dangerous conflict and the one teams most frequently ignore. The experiment increases immediate conversions but decreases retention or customer satisfaction.

Behavioral explanation: the change probably increased urgency or reduced friction in a way that generated low-quality conversions — users who would not have converted organically and who are less likely to retain.

Action: Do not ship. The long-term metric is almost always more important than the short-term metric. If you must optimize short-term conversion, find a way to do it without degrading retention.

Primary Metric Flat, Secondary Metric Improved

The experiment did not move the primary metric, but a secondary metric improved significantly.

This is useful learning but not a ship decision. The secondary metric improvement suggests the change had a real effect on user behavior — it just did not translate to the outcome you care most about. Investigate the gap between the secondary metric and the primary metric. There may be a bottleneck between them that a follow-up experiment can address.

Guardrail Metric Degraded Slightly

The primary metric improved, but a guardrail metric degraded just beyond the predefined threshold.

This is a judgment call, but the default should be to not ship. Guardrails exist for a reason. If you override them when they are inconvenient, they lose their purpose.

Instead, investigate whether the guardrail degradation is mechanically linked to the primary metric improvement (a tradeoff) or independent (a bug or side effect that can be fixed).

Metrics Move in Different Directions for Different Segments

The overall result is flat, but the experiment helped one segment and hurt another.

This is actually one of the most valuable experiment outcomes. It tells you that different users have different needs. Consider:

• Can you target the change only to the segment that benefits?
• Does the segment analysis suggest a more nuanced hypothesis?
• Is the segment difference driven by a confound (like device type or geography) rather than a real behavioral difference?

Building a Multi-Metric System

The Metric Dictionary

Create a shared document that defines every metric your team uses in experiments:

• Exact calculation (including edge cases like how nulls are handled)
• Data source and known limitations
• Historical baseline and variance
• Whether it is typically used as primary, guardrail, or secondary
• Known correlations with other metrics

Metric disagreements often stem from people defining the same metric differently. A shared dictionary eliminates this.

The Correlation Map

Build a correlation matrix of your key metrics using historical data. Understanding which metrics move together — and which move independently — helps you predict metric conflicts before they happen and interpret them when they do.

If two metrics are highly correlated, improving one should improve the other. If they move independently, they can diverge without indicating a problem. If they are negatively correlated, improving one at the expense of the other may be unavoidable — and you need to decide which matters more.

Regular Metric Reviews

Quarterly, review your metric framework:

• Are the guardrail thresholds still appropriate?
• Have new metrics become available that should be tracked?
• Have any metrics become unreliable due to instrumentation changes?
• Do the metric correlations still hold, or have they shifted?

Metric systems need maintenance just like code systems. Neglecting them produces decisions based on stale or inaccurate signals.

FAQ

How many metrics should I track per experiment?

One primary, two to four guardrails, and as many secondary/observation metrics as you want. The key constraint is that only one metric drives the ship decision.

What if stakeholders disagree on which metric should be primary?

This is a strategy conversation, not a data conversation. The primary metric should reflect the business objective for the current quarter. If stakeholders cannot agree on the objective, that disagreement needs to be resolved before any experiments are designed.

Should I use composite metrics to combine multiple outcomes?

Composite metrics can work but are risky. They mask important dynamics (one component improving while another degrades). If you use them, always decompose the composite in your analysis to understand what is driving the overall result.

How do I prevent post-hoc metric shopping?

Pre-registration. Document all metrics, their roles (primary, guardrail, secondary), and the decision framework before the experiment launches. Make this document visible to all stakeholders.