experiment-results-interpreter

Overview

The ab-test-validity-checklist skill confirms that an experiment was run correctly - clean SRM, honest peeking discipline, pre-declared OEC. This skill covers the next question: given a valid experiment, what does the result actually mean, and is it safe to ship?

The two most common failure modes at this stage, per Kohavi, Tang, Xu Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020, ISBN 9781108724265), are:

1. Shipping a result that is statistically significant but not practically meaningful.
2. Shipping a result that will not persist because it reflects novelty, primacy, or interaction artefacts rather than genuine long-term user value.

This skill is a pure reference for data scientists and PMs reading final experiment dashboards.

When to use

• Reading the results of an experiment that has reached its pre-declared sample size or end date.
• Preparing the ship/no-ship decision document.
• Auditing a past ship decision that produced unexpected post-launch outcomes.
• Coaching a PM or analyst who is over-indexing on p-values or under-checking guardrails.

How to use

Work through the six interpretation steps in order. Each step has a hard stop: if a step blocks, do not proceed to the next.

Step 1 - Practical significance before statistical significance

A statistically significant effect can still be too small to matter: results "may be statistically significant but not practically significant" (Nielsen Norman Group, sources).

The minimum practically significant effect (MPSE) must be declared in the pre-registration (proposal.yml). At read-time:

A statistically significant result with a point estimate well below the MPSE is a no-ship unless the maintenance cost is zero and the direction is consistent with strategy.

Step 2 - Confidence intervals, not just p-values

A 95% confidence interval gives the plausible range of the true effect - "both directionality and magnitude of effects simultaneously" (Statsig, sources) - which a binary p-value cannot.

Reading a result:

Width matters. A narrow CI means high power and a precise estimate; a wide CI means the experiment was underpowered, so extending runtime or pooling more traffic will narrow it. Variance-reduction techniques (CUPED) narrow the CI without inflating the false-positive rate (Microsoft ExP, sources) - prefer platforms that apply it by default.

Do not convert CI edges back to p-values to decide - the CI is the complete picture.

Step 3 - Novelty and primacy effects

A statistically and practically significant result in week 1 may not persist. Two opposing artefacts corrupt early-period estimates:

Novelty effect: users react to the mere newness of a change; engagement (clicks, session length) inflates above the true long-run level, then decays.

Primacy effect (resistance to change): new UI or workflows initially hurt task-completion because users must relearn existing habits; the treatment appears worse early, then improves as users adapt. Kohavi et al. note novelty and primacy are "significant causes of treatment effects changing over time but are not the sole causes" (sources).

Detection and mitigation:

Microsoft ExP found "14-day surprises" - the second week's estimate falling outside the first week's 3-sigma CI - at roughly 4% of experiments, far more than the theoretical rate (sources). Minimum run time: two full weeks before drawing ship conclusions from experiments that change UI patterns; feature launches with no UX learning curve may need only one week.

Step 4 - Interaction effects

An experiment running concurrently with other experiments may have its treatment effect inflated, deflated, or reversed by interference.

Two types:

Between-experiment interaction: variant A of experiment X and variant B of experiment Y are assigned to overlapping user populations. If the two treatments interact (positively or negatively), the OEC measured for X is partly caused by Y's presence (Microsoft ExP, "A/B Interactions: A Call to Relax", sources).

Treatment spillover: a social or marketplace product where treating some users changes outcomes for untreated users in the same experiment (network effects). The control group is contaminated; the measured effect is attenuated. Kohavi et al. (ISBN 9781108724265) categorise this as a stable unit treatment value assumption (SUTVA) violation.

Detection checklist:

If a significant interaction is identified, the measured effect is confounded. Options: (a) isolate with mutual exclusion and re-run, (b) include the interaction term in a factorial model, (c) block ship pending analysis.

Step 5 - Simpson's paradox

The aggregate OEC lift may be positive while every segment shows a negative or neutral lift - or vice versa. This is Simpson's paradox.

Simpson's paradox is when "a trend appears in several groups of data but disappears or reverses when the groups are combined" (Wikipedia, sources).

In A/B testing, it surfaces when:

• Traffic allocation differs across segments (e.g., mobile gets 60% of control but 40% of treatment due to a holdout policy or ramp-up).
• The OEC baseline differs strongly across segments (e.g., power users convert at 12%, new users at 2%).
• The treatment effect differs in sign across segments.

Detection:

For each major segment (device, country, user-cohort, new vs returning):
  1. Compute per-segment OEC lift and CI.
  2. Verify direction is consistent with the aggregate result.
  3. Check that per-segment traffic allocation matches the overall ratio.

If direction flips in a large segment: the aggregate result is misleading. Segment-level results are the truth; the aggregate is an artefact of unequal allocation. Do not ship on a positive aggregate with a negative segment that represents > 20% of users.

The ab-test-validity-checklist Step 7 includes a "segment-stability" gate for this reason - this skill provides the interpretive depth behind that gate.

Step 6 - Guardrail check before ship

Per guardrail-metrics-reference: no ship decision is valid without confirming that no guardrail metric has breached its block threshold.

Ordered check:

1. Load the guardrail dashboard for the experiment.
2. For each declared guardrail metric:
   a. Is the observed change within the alert threshold? (investigate, but not blocked)
   b. Does the observed change breach the block threshold? (STOP - no-ship)
3. If any guardrail is on alert: document the finding and make an
   explicit call (accepted risk + rationale OR extend experiment).
4. If all guardrails are within alert thresholds: proceed to ship.

Common guardrail check failures before ship:

Measuring only the OEC can "disregard important information" (Nielsen Norman Group, sources) - always check guardrails alongside it.

Example

Scenario: Redesigned onboarding flow experiment. Declared OEC: 7-day activation rate. MPSE: +0.5pp absolute. Alpha 0.05, 80% power. Ran 14 days. Result reads: lift = +1.2pp (95% CI: +0.3pp, +2.1pp).

Step 1 - Practical significance: Point estimate +1.2pp > MPSE +0.5pp. Lower CI bound +0.3pp is below MPSE. Minimum realistic effect is 0.3pp - marginal. Discuss with product whether 0.3pp justifies the complexity.

Step 2 - CI read: CI entirely above zero; statistically significant. Width (1.8pp) is moderate. Acceptable - not underpowered.

Step 3 - Novelty check: Week 1 lift was +2.1pp; week 2 lift was +0.9pp. Declining trend. Novelty effect likely inflating week 1. Use week 2 estimate (+0.9pp) as the stable-state estimate - still above MPSE.

Step 4 - Interaction: No other experiment running in onboarding funnel. SUTVA: metric is per-user (not network). Clear.

Step 5 - Simpson's: Mobile segment (45% of traffic): lift +0.7pp. Desktop segment (55%): lift +1.6pp. Directions consistent; no paradox.

Step 6 - Guardrails: API p95 latency +3% (alert threshold +5% or +50ms block). Within alert, no breach. DAU stable. Opt-out rate flat. All green.

Ship decision: Ship, citing week-2 stable estimate +0.9pp and clean guardrails. Document novelty decay in the ship note.

Anti-patterns

Limitations

• This skill does not validate the experiment harness. SRM, telemetry, and peeking discipline belong in ab-test-validity-checklist. This skill assumes the harness is valid.
• Novelty / primacy detection requires two or more weeks of data. Products under launch pressure may not have it.
• Interaction effects are hard to detect without factorial design. Audit-based detection (Step 4) is heuristic, not conclusive.
• Simpson's paradox analysis surfaces the descriptive pattern, not the causal explanation. Requires domain knowledge to resolve.
• Guardrail thresholds are pre-declared. Post-hoc threshold adjustment to clear a guardrail is p-hacking; this skill cannot prevent that without governance enforcement.

References

Full source attributions and verbatim quotes: references/sources.md. Primary source is Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020, ISBN 9781108724265); companion catalogs ab-test-validity-checklist, guardrail-metrics-reference, peeking-problem-reference.