Guardrail metrics

A guardrail metric is a measure that must not significantly degrade for an experiment to ship, even if the primary metric (the OEC - Overall Evaluation Criterion) improves. The guardrail prevents "we shipped 5% revenue improvement, but latency 30% worse and we discovered too late." Per Kohavi et al. Trustworthy Online Controlled Experiments (Cambridge Univ. Press, ISBN 978-1108724265), this is "the most important class of metrics after the OEC."

This reference is consumed by ab-test-validity-checklist and the SDK harnesses in experiment-sdk-testing.

How to use this reference

1. Pick the guardrail set for the experiment's surface (web / API / mobile / revenue / trust) from the canonical-set table.
2. Declare each guardrail - metric, direction, block threshold - in the experiment config before launch (see the pre-commitment worked example below).
3. Set two-tier levels per metric (alert + block), using max(%, absolute) on fast endpoints - see thresholds-and-corrections.md (opens in new window).
4. Correct for multiple comparisons across the OEC + N guardrails (Bonferroni / FDR) - see thresholds-and-corrections.md (opens in new window).
5. Gate the ship: block if any guardrail crosses its pre-declared block threshold, even when the OEC wins.

When to use

• Designing the metric set for a new experiment.
• PR review of experiment config changes.
• Pre-ship review: did we have guardrails on the right things?
• Investigating "we shipped X but Y broke" incidents.

The guardrail taxonomy

Four classes:

Microsoft's Experimentation Platform team writes that "we're always warning our customers to be vigilant when running A/B tests" and that "we warn them about the pitfalls of even tiny SRMs (sample ratio mismatches)" (A/B Interactions: A Call to Relax (opens in new window)). Tiny SRMs (per the peeking.md (opens in new window) sibling concept) and degraded guardrails are the canonical ship-and-regret sources.

The OEC vs guardrail relationship

• OEC - the metric you want to improve (e.g., revenue, signups, retention).
• Guardrail - the metric you don't want to break (e.g., latency, error rate).
• Driver - intermediate metric that explains why OEC changes (e.g., click-through rate explains conversion).

Per Kohavi et al.: the OEC is one metric (or a weighted combination), declared in advance, with a power calculation. The guardrails are the rest of the dashboard - short-term loss is acceptable if within bounds, but a significant degradation blocks ship.

Standard guardrails - the canonical set

Per Kohavi et al.: always include a quality guardrail (latency / error) - the most-missed category in real experiments.

Pre-commitment vs post-hoc

Guardrails must be declared before the experiment starts. Per Kohavi et al.: post-hoc guardrails are p-hacking - if you look at 50 metrics, some will spuriously fail.

Worked example - declare every guardrail in the experiment config before the experiment starts:

experiment: feed-ranking-v3
oec: ctr_per_session
power:
  primary_metric: ctr_per_session
  expected_effect: +1.5%
  alpha: 0.05
  beta: 0.20
guardrails:
  - metric: api_p95_latency
    direction: not-increase
    block_threshold: +10% or +50ms
  - metric: dau
    direction: not-decrease
    block_threshold: -1%
  - metric: error_rate
    direction: not-increase
    block_threshold: +0.1pp absolute

The block_threshold on api_p95_latency uses the max(%, absolute) rule so a fast endpoint can't ship a small absolute regression that a percentage alone would miss.

Anti-patterns

Limitations

• Guardrails are negative-defined. They prevent ship-and-regret; they don't measure success.
• Latency-as-guardrail interacts with caching. A cache hit rate shift changes apparent latency without product impact.
• Engagement guardrails are noisy. DAU varies with day-of- week, seasonality. Require longer experiments to surface signal.
• Pre-commitment is hard to enforce. Code review of experiment configs is the only practical gate.
• Guardrail-only dashboards miss the bigger picture. Pair with a counterfactual analysis dashboard.

Setting thresholds and correcting alpha - deep reference

Once the guardrail set is declared, the quantitative evaluation rules live in one companion reference:

• Thresholds + multiple-comparison correction - per-metric alert/block levels (with the max(%, absolute) rule) and Bonferroni / Benjamini-Hochberg correction across the OEC + N guardrails: thresholds-and-corrections.md (opens in new window).

References

• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge University Press, 2020). ISBN 978-1108724265.
• Microsoft Experimentation Platform: microsoft.com/en-us/research/group/experimentation-platform-exp/ (opens in new window).
• Quantitative evaluation mechanics (with their citation): thresholds + multiple-comparison correction in thresholds-and-corrections.md (opens in new window).
• Companion: peeking.md (opens in new window); design gates in ab-test-validity-checklist.
• SDK harnesses: experiment-sdk-testing.

The peeking problem

In classical (fixed-horizon) hypothesis testing, the test is run once, on a pre-declared sample size, at a pre-declared alpha (typically 0.05). Looking at the data and stopping when significance is reached before the pre-declared end inflates the false-positive rate well above alpha - sometimes to 30%+ at naive 0.05.

This is the peeking problem. Per Kohavi et al. Trustworthy Online Controlled Experiments (ISBN 978-1108724265): "Repeated significance testing is one of the most common mistakes in practical A/B testing."

This reference is consumed by ab-test-validity-checklist and the sample-ratio-mismatch-detector agent.

How to use this reference

1. Choose the analysis regime up front: fixed-horizon (no peeking) or a peek-protected method (sequential alpha-spending or always-valid / mSPRT) - see the three corrections below.
2. If you must stop early, pick a sequential schedule (Pocock vs O'Brien-Fleming) or mSPRT, and pre-declare it before launch.
3. Lock the look schedule: hourly dashboards and "where are we now?" checks are not decision looks - only the pre-declared schedule is.
4. Read the peek-protected p-value at the gate, never the naive mid-experiment p-value.
5. Stack corrections: if the experiment also has guardrails, combine the peeking correction with the guardrail (Bonferroni / FDR) correction - see the worked example below.

When to use

• Designing the stop-early policy for an experiment platform.
• Auditing an "early ship" decision - was the math valid?
• PR review of a new experiment dashboard / analysis flow.
• Investigating "we shipped, then the effect disappeared."

Why naive peeking inflates false positives

At alpha=0.05, the test is calibrated to give a 5% false-positive rate if you look once. If you look every day for 30 days and ship at the first significance - at each look, the test has a fresh chance to spuriously hit. The total false-positive rate compounds.

Per Microsoft Experimentation Platform research (microsoft.com/en-us/research/group/experimentation-platform-exp/ (opens in new window)): common patterns that surface this - dashboards that update hourly, "early-stop" buttons in experimentation UIs, manager asks for "where are we now?" mid-experiment.

Three corrections

1. Fixed-horizon test (pre-declared)

Decide N in advance via power analysis; collect N samples; do one test; ship or not. No peeking, no early stop.

Pros: standard p-value interpretation, full alpha budget on the declared test.

Cons: must wait for N. Cannot stop early on obvious winners (opportunity cost) or obvious losers (continuing risk).

2. Sequential testing with alpha-spending

Pre-commit to multiple looks, each with a fraction of the alpha budget. Two canonical schedules:

Implementation: declare K looks in advance; at each look k, the rejection threshold is computed from the cumulative alpha spent (per the schedule). If the test stat exceeds the threshold, stop.

Math: Σ alpha_k = alpha_total.

3. Always-valid inference / mSPRT

Per Johari, Pekelis, Walsh "Always Valid Inference" (paper ID: arXiv:1512.04922) and related work, the mixture sequential probability ratio test (mSPRT) lets you peek arbitrarily often without inflating alpha. The trade-off: less powerful per sample than fixed-horizon.

This is the foundation of "valid sequential" experimentation in Optimizely / Statsig / similar - they expose p-values that are always valid under continuous monitoring.

Per Optimizely's sequential-testing docs (a derivative of mSPRT): the platform allows the user to look at any time; the p-value remains valid.

Visual intuition

Decision boundary in tests

Tests for an experimentation platform must verify:

Worked example - stacking peeking + guardrail corrections

Per guardrails.md (opens in new window): the guardrail-correction (Bonferroni / FDR) stacks with the peeking correction. Don't apply only one if both are needed.

For an experiment with one OEC, 10 guardrails, and 5 looks:

The strict math is rarely applied this thoroughly; pragmatically most platforms apply sequential + per-metric alpha but not formal multi-comparison correction across guardrails.

Anti-patterns

Limitations

• Always-valid inference is less powerful. Same effect size requires more samples than fixed-horizon. Trade convenience for sample efficiency.
• Sequential methods require pre-declared schedules. The alpha-spending isn't "fluid"; the schedule is fixed in advance.
• Multiple-testing correction across many metrics is brutal. Per-metric alpha after Bonferroni × 20 metrics = 0.0025.
• Operator behaviour is the real bottleneck. Math is robust to peeking; humans are not. Education + UI gating matter.
• Doesn't help with novelty / primacy effects. Statistical validity doesn't fix "users react to change, then revert."

References

• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020). ISBN 978-1108724265, ch. on sequential testing.
• Johari, Pekelis, Walsh. Always Valid Inference: Continuous Monitoring of A/B Tests (arXiv:1512.04922).
• Microsoft Experimentation Platform: microsoft.com/en-us/research/group/experimentation-platform-exp/ (opens in new window).
• Companion: guardrails.md (opens in new window); design gates in ab-test-validity-checklist.
• SDK harnesses: experiment-sdk-testing.

Sources and verbatim citations

Full source attributions for the interpretation steps in SKILL.md. Each inline [sources](references/sources.md) link points here.

Practical vs statistical significance (Step 1)

Nielsen Norman Group, A/B testing guide: results "may be statistically significant but not practically significant" - a test can show reliable differences that lack meaningful business value. nngroup.com/articles/ab-testing/ (opens in new window)

Confidence intervals (Step 2)

Statsig confidence-interval docs: "A 95% confidence interval should contain the true effect 95% of the time"; it is "an intuitive way to quantify the uncertainty" that gives "both directionality and magnitude of effects simultaneously." docs.statsig.com/experiments/statistical-methods/confidence-intervals (opens in new window)

Microsoft ExP variance-reduction research: CUPED and similar techniques produce "narrower confidence intervals, with values that are closer to the estimated effect" without sacrificing the false-positive rate. microsoft.com/en-us/research/group/experimentation-platform-exp/articles/deep-dive-into-variance-reduction/ (opens in new window)

Novelty and primacy effects (Step 3)

Kohavi, Tang, Xu, Trustworthy Online Controlled Experiments (Cambridge Univ. Press, 2020, ISBN 9781108724265): "novelty and primacy effects are significant causes of treatment effects changing over time but are not the sole causes."

Microsoft ExP external-validity research: "14-day surprises" - the second week's estimate falling outside the first week's 3-sigma confidence interval - occurred at roughly 4% of experiments, far more than the theoretical rate. microsoft.com/en-us/research/group/experimentation-platform-exp/articles/external-validity-of-online-experiments-can-we-predict-the-future/ (opens in new window)

Wikipedia, novelty effect: a temporary boost from "introducing new elements on some activity or behavior" rather than underlying improvement. en.wikipedia.org/wiki/Novelty_effect (opens in new window)

Interaction effects (Step 4)

Microsoft ExP, "A/B Interactions: A Call to Relax": addresses A/B interactions in concurrent experiment design alongside the pitfalls of even tiny SRMs. microsoft.com/en-us/research/group/experimentation-platform-exp/articles/ (opens in new window)

Kohavi et al. (ISBN 9781108724265) categorise treatment spillover as a stable unit treatment value assumption (SUTVA) violation.

Simpson's paradox (Step 5)

Wikipedia, Simpson's paradox: "a trend appears in several groups of data but disappears or reverses when the groups are combined." Canonical Berkeley admissions example: men were admitted at higher aggregate rates (44% vs 35%) while women had better odds in most individual departments, because women applied to more competitive departments. en.wikipedia.org/wiki/Simpson%27s_paradox (opens in new window)

Guardrails (Step 6)

Nielsen Norman Group: "if you measure only one metric to determine whether your test is successful, you might disregard important information." nngroup.com/articles/ab-testing/ (opens in new window)

Companion catalogs

ab-test-validity-checklist; sibling references peeking.md (opens in new window) and guardrails.md (opens in new window).

Guardrail thresholds and multiple-comparison correction

Deep reference for guardrails.md (opens in new window). Consult when setting per-metric alert/block levels and correcting alpha across the OEC + N guardrails.

Setting guardrail thresholds

A guardrail typically has two levels:

Example for API latency p95:

The "whichever is greater" handles fast endpoints where 10% is trivially small in absolute terms.

Multiple-comparison correction

With one OEC + N guardrails (typically 10-20), a fixed-alpha significance test means you'll see N×0.05 false positives on average. Per Kohavi et al., apply Bonferroni or Benjamini- Hochberg correction:

Reference

• Kohavi, Tang, Xu. Trustworthy Online Controlled Experiments (Cambridge University Press, 2020). ISBN 978-1108724265.