A variant won an A/B test, became the default, and stayed there. Months later, traffic comes from different campaigns, more shoppers use phones, and the offer has changed. Should you rerun the experiment?
Review the original decision when a material assumption changes. Retest when the relative value of the two experiences is uncertain, the decision matters, and a reliable comparison is feasible. Repair broken behavior first. A falling conversion rate alone does not establish that the old winner stopped helping.
Use calendar reminders to review the evidence. Let that review determine whether another experiment is worth running.
This guide gives ecommerce and CRO teams a practical policy for choosing between repair, investigation, and a fresh experiment. The review record and examples are illustrative recommendations, not a universal expiry schedule for test results.
Recover the decision behind the winner
Find the original experiment brief, code revision, targeting, dates, primary outcome, uncertainty, and guardrail findings. Record what customers actually saw and what the comparison established. “Green banner won” is less useful than “making delivery eligibility visible near the buying controls improved purchases among the tested visitors.”
Check how the winner was selected. Research by Deng, Li, and Guo explains that choosing the best observed treatment from multiple candidates can bias its estimated effect upward. Their work studies selection followed by validation; it does not prescribe a calendar for retesting shipped designs. An unusually large original estimate deserves scrutiny before it becomes a permanent business forecast. Treatment selection and validation research.
If the original analysis is missing or flawed, label the decision's evidence uncertain. That is a different problem from a trustworthy finding becoming less applicable to today's audience.
Use three checks before spending traffic
First, inspect implementation. Is the intended experience still present on eligible pages? Does it preserve product selection, accurate prices, links, and purchase behavior? A theme update that removes the delivery line is an implementation defect. Restore an approved working experience and verify it through the normal release process.
Second, inspect measurement. Did event definitions, consent behavior, identity rules, or attribution windows change? Compare application records with analytics using approved test transactions. If the metric changed meaning, repair reporting before interpreting the trend.
Third, inspect context. Did the audience, offer, inventory, device mix, or surrounding journey change in a way that could affect the reason the variant helped? Write the proposed mechanism explicitly. “More mobile visitors” is a starting observation; “the expanded information block now competes with the size selector on small screens” is a testable concern.
| Finding | Next action |
|---|---|
| Reproducible blocked purchase or inaccurate promise | Repair or roll back through the incident process. |
| Missing events or changed metric definition | Investigate instrumentation and restore comparability. |
| Material context change with uncertain relative performance | Consider a new randomized comparison. |
| No relevant change and limited decision value | Keep the experience and record the next review trigger. |
The A/B testing QA checklist supplies the implementation checks. Use this policy to decide whether another experiment is warranted after those checks pass.
Separate audience mix from treatment performance
A sitewide conversion decline can happen even when conversion within each audience group stays unchanged. Here is a hypothetical example of the current shipped experience across two periods, with no concurrent alternative being tested.
Each period contains 10,000 distinct eligible visitors, complete seven-day purchase follow-up, and two mutually exclusive acquisition groups defined at entry. Conversion means visitors with at least one qualifying purchase divided by eligible visitors. Assume identical purchase definitions and stable group-specific rates.
| Acquisition group | Earlier visitors | Earlier purchasers | Later visitors | Later purchasers |
|---|---|---|---|---|
| Group A: 4% purchase rate | 8,000 | 320 | 2,000 | 80 |
| Group B: 1% purchase rate | 2,000 | 20 | 8,000 | 80 |
| Total | 10,000 | 340 | 10,000 | 160 |
Overall conversion falls from 3.4% to 1.6%, a decrease of 1.8 percentage points, despite unchanged rates within both groups. The mix changed. These invented numbers demonstrate arithmetic, not a customer result or a statistical finding.
This comparison cannot tell us whether the alternative experience would perform better today. It contains no current control group. Segmenting the dashboard helps identify questions; it does not recover the missing counterfactual.
When using GA4 to investigate acquisition, keep report scope consistent. Google distinguishes user-scoped acquisition dimensions from session-scoped traffic acquisition dimensions. Switching between them can change which channel receives a visitor's activity. Google's acquisition-report comparison.
For experiment analysis, use the assignment population and acquisition definition in the analysis plan. A session dashboard and a visitor-randomized test need not have matching denominators.
Define review triggers around the mechanism
Give each shipped decision an owner and a review trigger. Useful triggers include replacing the tested offer, moving the affected control, launching a substantially different acquisition campaign, or entering a season whose purchase journey differs from the original test.
A calendar reminder can prompt that review. It should not automatically consume another experiment slot. Choose its frequency according to change frequency, decision importance, and team capacity. Treat numerical traffic-shift thresholds as local operating choices that need justification, rather than borrowed standards.
Use this compact record:
| Field | What to record |
|---|---|
| Original decision | Experience adopted, comparison, evidence, and uncertainty. |
| Expected mechanism | Why the change should help the intended audience. |
| Original context | Offer, traffic, devices, inventory, dates, and page revision. |
| Review trigger | Specific change that could weaken that explanation. |
| Current evidence | Observations, implementation checks, and measurement limitations. |
| Decision at stake | What the team would change after a useful result. |
| Owner and disposition | Repair, investigate, retest, or retain; next review condition. |
An illustrative note could read: “The original test moved delivery details beside the purchase button. A new mobile campaign changes the viewing context. QA and tracking checks pass, but customer observations suggest the expanded block competes with size selection. Review a compact alternative if the planned comparison can finish before our campaign decision.” This is a proposed new presentation comparison; it does not establish that the original treatment stopped helping.
Design the retest for today's decision
Compare the current experience with a clearly specified alternative under current conditions. If you need to know whether the original treatment still helps, preserve other meaningful differences between the two arms. If you instead compare two newly redesigned pages, describe the result as a new design comparison.
Do not restore obsolete prices, inaccurate delivery promises, or a known broken interaction to recreate an old control. Use an approved valid alternative and document how the question differs from the original experiment.
Before exposure, specify eligibility, randomization unit, stable assignment, primary metric, guardrails, attribution window, sample requirements, and the analysis method's stopping rule. Plan how prior exposure and returning visitors will be handled with the experiment analyst. Do not simply pool the old and new experiments: they may estimate effects in different contexts.
GOV.UK's A/B testing guidance describes a control, variation and randomized comparison. Apply those design basics to the current decision rather than treating the old result as today's control group.
If purchases are the primary outcome, monitor commercial consequences too. The revenue-per-visitor guide explains why extra purchases can coexist with poorer revenue or contribution.
If traffic cannot support the decision, record the uncertainty and use customer research and operational evidence to prioritize improvements. Do not relabel a before-and-after trend as a randomized result.
Keep the implementation and evidence together
For a presentation retest, prepare a narrow build request: move the approved delivery explanation near the existing buying controls while preserving prices, eligibility, product validation, destinations, and analytics hooks. Review difficult states and repeat QA in the destination testing platform.
For teams with access, the VibeVariant getting-started guide explains selecting page elements, refining editable JavaScript and CSS, previewing in the attached tab, and manually copying reviewed code into an experimentation platform. Public installation is not open yet; the guide offers a scripted example. The AI A/B testing guide explains how to keep that implementation bounded by the approved brief. Assignment, traffic splitting, statistics, and outcome measurement remain in your stack. Repeat destination QA; a preview does not establish production reliability or conversion benefit.
Keep the code, review record, and eventual result together. Microsoft's long-term experiment research identifies cookie stability, selection bias and survivorship bias as interpretation risks. If you consider an ongoing holdout, ask the analyst how its design handles those risks.
Start with one important shipped decision. Recover its evidence, check its implementation and measurement, and write the condition that would make you reconsider it. Run a retest when its answer can change what you ship.
