ab-test-validity-checklist

Overview

This skill produces the pre-flight + post-flight validity checklist for an A/B test. Each item is a gate; failing one without explicit acknowledgment invalidates the experiment.

Per Kohavi et al. Trustworthy Online Controlled Experiments (ISBN 978-1108724265): "More than 50% of experiments in practice are invalidated by issues the checklist catches."

The output: a per-experiment markdown checklist + a sign-off form for the experiment owner.

When to use

• Launching a new experiment.
• Auditing an experiment that produced surprising results.
• Building experimentation governance / a peer-review process.
• PR review of experiment configuration changes.

Step 1 - Pre-registration

Document before launch:

Candidate guardrail set to pick from, with typical block thresholds (per guardrail-metrics-reference; Kohavi et al. Trustworthy Online Controlled Experiments, ISBN 978-1108724265):

Each guardrail carries two levels: alert (statistically significant degradation - investigate before ship) and block (past the pre-declared limit - ship-decision flips to "no"). Per Kohavi et al., always include a quality guardrail (latency / error); it is the most-missed category. Percentage-only thresholds on fast endpoints are a trap - use max(%, absolute).

Commit this to the repo as experiments/<id>/proposal.yml. Any post-launch change requires explicit team approval.

Step 2 - Sample Ratio Mismatch (SRM)

Per Microsoft Experimentation Platform research (KDD 2019 paper "Diagnosing Sample Ratio Mismatch in Online Controlled Experiments"): if the observed allocation (e.g., 50.3% A, 49.7% B) deviates significantly from intended (50% / 50%), the experiment is invalid until root cause is found. SRM signals:

• Logging bugs (assignments not all logged)
• Bot filtering (different ratios filtered per arm)
• Redirects (one variant redirects more)
• Telemetry drops (one variant has heavier client → more drops)
• Randomisation bugs (hash collisions)

Chi-square test:

χ² = Σ ((observed_i - expected_i)² / expected_i)

For 2 arms at 50/50 with N=1e6 users:

• Expected: 500k each
• Observed: 503k / 497k
• χ² = (3000²/500000) + (3000²/500000) = 36
• p-value: < 0.0001

Threshold: p < 0.0001 is the canonical SRM-detection boundary (the chi-square is super-sensitive at large N; this threshold prevents false-positive SRM alarms).

If SRM is detected: stop ship discussion; root-cause first.

Step 3 - Assignment integrity

Tests for the assignment SDK / service:

These tests live in the SDK-specific test skills per statsig-test, optimizely-test, etc.

Step 4 - Telemetry correctness

Verify the event firing matches the proposal:

• Conversion events fire exactly once per user per conversion-eligible session.
• Exposure events fire for everyone who could see the variant (not just those who actually saw it - that's a different measure, "treatment effect on the treated").
• Guardrail metrics are queryable against the experiment partition (variant ID joined to event stream).

Step 5 - Peeking discipline

Per peeking-problem-reference:

Step 6 - Novelty / primacy effects

Per Kohavi et al.: users react differently to novel UX. Novelty inflates the early-period effect; primacy depresses it. Mitigation:

• Run for ≥ 2 weeks (typical mature-effect period).
• Segment results by "first exposure vs returning to treatment."
• For long-running tests, segment by week to spot trend reversal.

Step 7 - Post-experiment validation

Before ship:

Document each pass in experiments/<id>/result.md with the specific numbers.

Step 8 - Emit the checklist

The output of this skill: a markdown checklist + sign-off form.

# Experiment <id> - Validity Checklist

## Pre-registration (signed by: <owner>, date: <YYYY-MM-DD>)

- [ ] OEC declared: <metric>
- [ ] Power calc: N=<X>, alpha=0.05, beta=0.20, MDE=<Y>%
- [ ] Guardrails declared: <list with thresholds>
- [ ] Randomization unit: <user_id / device_id>
- [ ] Allocation: <50/50>
- [ ] Look schedule: <Pocock 5 looks at days 2,4,7,10,14>
- [ ] Stop-early rules: <on OEC reaching alpha-threshold>

## During experiment

- [ ] SRM check: chi-square p > 0.0001 ([result: <p>])
- [ ] Assignment integrity tests passing
- [ ] Telemetry validated

## Post-experiment (signed by: <reviewer>, date: <YYYY-MM-DD>)

- [ ] Pre-registration honoured (no scope changes)
- [ ] SRM final check: p > 0.0001 ([result])
- [ ] OEC significant (p=<X>; method: <Pocock>)
- [ ] All guardrails within thresholds:
   - api_p95_latency: +<X>% / +<Y>ms - <status>
   - dau: <X>% - <status>
- [ ] Multiple-comparison adjusted (method: <Bonferroni / BH>)
- [ ] Novelty assessment: effect persists week 2+? <yes / no>
- [ ] Segment stability: direction consistent? <yes / no>
- [ ] Trust metric stable? <yes / no>

## Ship decision: <ship / no-ship / extend>

Reasoning: <one paragraph>

Sign-off: <name>, <date>

Anti-patterns

Limitations

• Checklist is necessary, not sufficient. Quality of the underlying telemetry + assignment logic matter.
• Multiple-comparison corrections are conservative. May reject real wins.
• Novelty assessment needs ≥ 2 weeks. Pressure to ship fast conflicts.
• Doesn't catch ecosystem effects. Cross-experiment interaction, carry-over, etc. require global statistics.

References

• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020). ISBN 978-1108724265.
• KDD 2019 paper "Diagnosing Sample Ratio Mismatch" (Microsoft Research).
• Microsoft Experimentation Platform articles: microsoft.com/en-us/research/group/experimentation-platform-exp/ (opens in new window).
• Companion catalogs: guardrail-metrics-reference, peeking-problem-reference.
• Operationalised: statsig-test, optimizely-test, vwo-test, amplitude-experiment-test.