What VWO Gives an Experimentation Team—and What It Cannot Decide

VWO can report a Bayesian probability, a frequentist significance level, a sequential recommendation, a fixed-horizon result, or lower-rigor dynamic evidence. It can add approvals, guardrails, sample-ratio-mismatch alerts, and experiment-conduct warnings.

That means the sentence “we ran the test in VWO” tells you almost nothing about the statistical decision.

The platform matters, but the customer team chooses—or inherits—the model, monitoring approach, thresholds, metrics, roles, and response to warnings. A strong tool can make good practice easier. It cannot turn an undefined business decision into a valid experiment.

VWO’s current help center presents the newest workflow under the Wingify name. I use VWO/Wingify when discussing the current documented platform and preserve VWO where that is how teams commonly know the product.

This is a platform analysis, not a company-team case study

Netflix, Google, Apple, and Booking.com document experimentation systems used inside or around their own products. VWO/Wingify sells experimentation capabilities to many customer organizations. Those customers can have radically different team structures and methods while using the same vendor.

The public documentation can establish what the platform supports:

• Bayesian and frequentist statistical models
• Sequential, fixed-horizon, and dynamic testing approaches
• Practical-equivalence regions and decision probabilities
• Multiple-comparison correction
• Campaign approvals and role permissions
• Data, conversion, minimum-runtime, guardrail, conduct, and SRM checks
• Web, feature, rollout, personalization, and multivariate workflows

It cannot establish what a particular customer selected, whether warnings were resolved, whether the primary metric was chosen before launch, or who owned the final decision.

This evidence boundary changes how program maturity should be assessed. VWO feature maturity belongs to the vendor. Experimentation maturity belongs to the customer operating the account.

What the platform owns—and what the team still owns

The important asymmetry is visible in every row: the platform can implement a declared policy, but it cannot supply the business rationale that makes the policy defensible.

A mature workflow starts before campaign creation

VWO/Wingify supports different product surfaces, so implementation details vary. Its feature experimentation workflow separates configuration, code implementation, and launch/monitoring. Feature tests require engineering support for SDK integration and flag checks; visual web tests may require less code.

The decision workflow should remain consistent:

1. Write the hypothesis, target population, primary metric, guardrails, and action rule.
2. Choose the smallest effect worth acting on and determine whether the available traffic can resolve it.
3. Select Bayesian or frequentist inference before the campaign starts.
4. Select sequential, fixed-horizon, or dynamic monitoring based on how the result will be used.
5. Apply a multiple-comparison correction when several variations compete with the baseline.
6. Configure targeting, traffic allocation, metrics, and exclusions.
7. Implement the treatment and exposure logic; perform technical QA.
8. Route the campaign through the required review and approval policy.
9. Launch and monitor Experiment Vitals without treating every chart movement as a decision.
10. Act only when the prespecified rule and data-quality conditions are satisfied.
11. Record the result, exceptions, guardrails, and final business decision.

If the team selects the model after seeing which output looks most favorable, the platform’s sophistication becomes a menu for outcome shopping. VWO/Wingify prevents one version of this problem by locking the statistical model after a campaign starts. That is a useful guardrail, but the team must still record why it chose the configuration.

SmartStats offers two models and three stopping approaches

The current SmartStats configuration documentation distinguishes the statistical model from the testing approach.

That is exactly the right separation. “Bayesian versus frequentist” describes how uncertainty is represented. “Sequential versus fixed horizon” describes when the evidence is evaluated and what repeated looks are allowed. They are related choices, not synonyms.

Bayesian is the default, but not the only model

The documented Bayesian model evaluates whether the improvement distribution crosses a Region of Practical Equivalence, or ROPE. Reports express the probability that a variation is better, worse, or equivalent to the baseline.

The frequentist option tests observed differences against a null hypothesis and reports significance levels. Teams can switch models while a campaign is a draft, but not after it is running or paused.

The Bayesian default is a product configuration—not evidence that every VWO customer has adopted a Bayesian decision policy. A team still needs to know the ROPE, threshold, prior behavior, minimum information, and stopping approach behind the recommendation. My Bayesian testing glossary explains why a probability statement is only as useful as the decision rule around it.

Sequential testing is for repeated looks

The sequential approach continuously evaluates results while adjusting thresholds for repeated analysis. The documentation says winner or disable recommendations appear when the corrected boundaries are crossed.

This is the appropriate family of methods when teams genuinely need to inspect evidence over time or stop early. It is different from looking at an ordinary fixed-horizon p-value every morning. See the sequential testing glossary for the peeking problem the correction is intended to solve.

Fixed horizon is for a committed single evaluation

Fixed Horizon calculates the required sample up front and evaluates after that visitor count is reached. It is the clean default for an ordinary decision when the team can commit to one final read and wants the design to cover full traffic cycles.

The method still requires a meaningful effect target, valid unit of analysis, power, and a realistic duration. The platform knowing a visitor count does not guarantee the event definition or commercial threshold is correct.

Dynamic mode is explicitly lower rigor

Dynamic testing has no fixed sample bound and continuously checks until the campaign is paused. VWO/Wingify’s own FAQ positions it for low-traffic cases where a team wants directional data quickly and can accept lower statistical rigor.

That is an honest product label. The correct use is exploratory prioritization or minimal-risk learning—not upgrading a noisy directional signal into a “decision-grade winner” because the business wants an answer.

Low traffic limits certainty; changing the dashboard mode does not manufacture information.

When a test is infeasible, use a sample-size calculator to expose the gap. Then choose a larger intervention, a higher-traffic metric closer to the mechanism, a longer horizon, a different research method, or an explicitly lower-stakes decision.

Multiple variations require a separate correction

Sequential correction addresses repeated looks. It does not automatically solve the increased false-positive risk from comparing several variations with one baseline.

SmartStats offers Bonferroni correction when a campaign has two or more variations, excluding the baseline. The correction tightens each comparison threshold and raises the required visitor count. VWO/Wingify recommends combining Bonferroni with sequential testing when teams both check frequently and test multiple arms.

Bonferroni is simple and conservative. A team should not disable it merely because the experiment becomes inconveniently large. The correct response may be to test fewer variants, prioritize stronger hypotheses, or choose a design whose multiplicity policy is justified in advance.

This illustrates a recurring program trade-off: more variants feel like faster learning, but each additional arm consumes traffic and creates another opportunity for a chance winner.

The report moves through five inference levels

The current reporting methodology describes five levels:

1. Empirical data: observed visitors, conversions, and total metric values.
2. Expected average: a distribution or interval for each variation’s underlying performance.
3. Expected improvement: the difference between variation and baseline distributions.
4. Probability or evidence: probability beyond the ROPE for Bayesian campaigns, or significance against the null for frequentist campaigns.
5. Decision: a recommendation after the configured threshold is crossed.

This hierarchy is useful because it stops readers from jumping directly from a raw conversion-rate difference to a winner label.

The documentation says the report remains in “Collecting Data” until the baseline has at least 500 visitors and one conversion, unless Observatory Mode changes that limit. That is a computation floor, not proof of adequate power. One conversion can make a statistic calculable while leaving it far too uncertain for a consequential decision.

When I review a VWO result, I ask for the statistical configuration and Experiment Vitals before interpreting the winner banner. The logo does not tell me the model, stopping rule, correction, ROPE, or whether the data passed basic integrity checks.

Experiment Vitals make data quality visible

VWO/Wingify’s Experiment Vitals continuously checks several failure modes:

• Missing or unusually low data tracking
• Missing conversion tracking
• Results viewed before a minimum runtime
• Negative guardrail movement
• Changes to metrics, allocation, targeting, or variation content during a live campaign
• Sample ratio mismatch between intended and observed allocation

These checks are valuable because statistical inference assumes the experiment was executed as designed. A perfectly calculated interval around corrupted exposure data remains corrupted.

The platform recommends at least seven days to cover weekly behavior and flags conclusions drawn earlier. Seven days is not a universal sufficient runtime. A test can cover a weekly cycle and remain underpowered; a high-frequency technical canary may support a different time scale with an appropriate sequential design.

The critical operating rule is what happens after a red vital. SRM, broken tracking, or an unplanned treatment change can invalidate causal interpretation. A mild guardrail alert may require risk review rather than automatic invalidation. The team should define those responses before launch instead of improvising when the preferred variation is ahead.

Approvals are available, but governance is optional and configurable

The Campaign Approvals documentation describes an Enterprise feature that lets creators send campaigns or unpublished live changes to designated approvers. Approvers can accept the campaign or request changes, and the activity timeline preserves feedback.

Only one of several designated approvers is required. Workspaces can also set time-based automatic approval or allow Developer API updates to bypass the workflow.

Those options are not inherently good or bad. They show that the product supports several governance models:

• Mandatory human review for high-risk campaigns
• Time-limited review for routine changes
• Direct API deployment for trusted automated workflows

The maturity question is whether the bypass policy matches risk. Enabling approvals and then auto-approving every campaign after a short timeout is not the same as expert review. Likewise, requiring a senior committee for every low-risk copy test can turn safety into delay without improving reasoning.

How decisions are made

VWO/Wingify can display a winner or disable recommendation after configured thresholds are crossed. The customer organization decides what the recommendation means operationally.

A complete decision record should include:

• The model and testing approach
• Threshold, ROPE or MDE, and multiplicity policy
• Primary metric and guardrails
• Experiment Vitals status and any violations
• Observed effect with interval or distribution
• Segments examined and whether they were prespecified
• Final action and named owner
• Any departure from the original rule

The platform can host parts of this record. The organization must still make exceptions visible and prevent a positive secondary metric from replacing a missed primary.

My experimentation method treats the output as decision-grade only when the plan, data health, analysis, and approval chain are complete. Otherwise the honest labels are directional, inconclusive, or cannot validate.

How mature is VWO—and how mature is its customer?

The current product documentation supports a feature-rich experimentation platform.

Buying the platform may raise the ceiling of a program. It does not establish where the customer operates beneath that ceiling.

Is VWO/Wingify right for your team?

The platform is a plausible fit when:

• The team needs packaged web or feature experimentation rather than building allocation and reporting infrastructure.
• Product, growth, engineering, and analytics can agree on metrics and roles.
• Traffic is sufficient for the effects the business cares about.
• The organization will configure a method once and respect its stopping behavior.
• Health checks and approvals will be tied to documented responses.

It is not a solution for:

• Insufficient traffic disguised as a statistics problem
• Broken event definitions or missing exposure data
• No named decision owner
• A culture that changes metrics after seeing results
• Complex marketplace interference or long-term causal questions that require a different design
• A repository problem when campaign results are never synthesized

Before buying or expanding the tool, inspect ten recent decisions. If the failures are mostly assignment, analysis compute, and deployment controls, platform capability may help. If the failures are vague hypotheses, irrelevant metrics, and unowned decisions, fix the operating model first.

FAQ

Is VWO Bayesian or frequentist?

It supports both in current SmartStats documentation. Bayesian is the default for new campaigns, but teams can select frequentist inference before launch. The model cannot be changed after the campaign starts.

Can teams look at VWO results every day?

They can inspect the interface, but valid decision behavior depends on the configured approach. Sequential mode adjusts for repeated looks. Fixed Horizon is intended for a committed evaluation after the planned sample. Dynamic is described as lower-rigor directional evidence.

Does VWO automatically correct multiple variants?

The platform offers Bonferroni correction and recommends it for two or more variations beyond the baseline. Whether it was enabled for a specific campaign must be verified.

Does a VWO winner banner mean the experiment should ship?

Not automatically. The team must confirm data integrity, the prespecified primary metric, practical effect, guardrails, decision risk, and any required approval.

Does using VWO indicate a mature experimentation program?

No. It indicates access to mature platform features. Program maturity depends on how the organization plans, configures, monitors, decides, and learns.

The debatable choice is whether a vendor should offer a lower-rigor Dynamic mode at all. Transparent directional labeling can help low-risk exploration; the danger is organizational pressure quietly turning that output into a winner. Governance, not interface wording alone, determines which outcome prevails.

If your VWO dashboard says “winner” but stakeholders still disagree, contact me with the plan, configuration, and decision at stake. I can help separate an evidence problem from a governance problem. To follow the complete series, subscribe to Lean Experiments.