chaos-drill-protocol

What this skill owns, and what it does not

This is the run protocol. It covers the window that opens when someone is about to inject a real fault into a running system and closes when the system has been confirmed back at baseline: what must be true before the fault starts (pre-flight gates), how wide it is allowed to reach (blast-radius bound), what is watched while it is live and what ends it early (sampling and abort criteria), and how recovery is confirmed rather than assumed.

Not owned here: designing the experiment. Choosing the steady-state hypothesis and its metric, choosing which real-world event to inject, and writing the experiment definition are a separate job that happens earlier. Per principlesofchaos.org, an experiment starts "by defining 'steady state' as some measurable output of a system that indicates normal behavior" and then hypothesizes "that this steady state will continue in both the control group and the experimental group." This protocol assumes a written hypothesis with a numeric threshold already exists. If it does not, stop and go do that first: a drill with no hypothesis produces a story, not a result. Also out of scope: aggregating results across drills into a trend, and the vendor-specific syntax of whichever injection tool is in use.

Why this protocol is strict

A chaos drill and an outage differ only by preparation. Google's own SRE account of a test-induced emergency describes a plan "to block all access to just one database out of a hundred" that within minutes made multiple dependent services inaccessible to external and internal users, because the team's assumptions about dependencies were incomplete and, in their words, they had not "tested our rollback procedures in a test environment" (sre.google/sre-book/emergency-response). Every gate below exists because skipping it is how that outcome happens.

The drill contract

Fill this in and get it agreed before anything is injected. Every field is a number or a name, never a judgment to be made later.

A drill missing any row is not ready to run. This mirrors how the Chaos Toolkit experiment format treats safety as structural rather than optional: the format has a first-class rollbacks property whose actions "attempt to put the system back to its initial state", and if the steady-state hypothesis is not met up front then "the Method element is not applied and the experiment MUST bail out" (chaostoolkit.org/reference/api/experiment).

Stage 1: pre-flight gates

These four are gates, not a checklist to feel good about. Each one is pass or fail. A single failure stops the drill. There is no partial pass, no "we will watch it closely instead", and no waiving a gate because a maintenance window is expiring. A failed gate is fixed and the drill is rescheduled.

Gate 1: the target is not production

Read the environment identity from the tooling itself (the active cluster or account context), not from what someone typed in the ticket. If the resolved context names production, the drill stops.

Running in production is a legitimate long-term goal, not a starting point. principlesofchaos.org does state that "Chaos strongly prefers to experiment directly on production traffic", and Netflix's Chaos Monkey "is responsible for randomly terminating instances in production" (netflix.github.io/chaosmonkey). Both describe mature programs with years of tooling behind them. This protocol gates on non-production and treats promotion as a separate decision made after repeated clean runs.

Gate 2: the baseline is healthy, and measured

Take a real baseline measurement immediately before injection. Not last week's dashboard screenshot: a fresh reading over a window long enough to be stable. Record the actual numbers for every signal the abort and recovery criteria reference, because those criteria are expressed relative to this baseline.

If the target is already degraded, the drill stops. The experiment method is to look "for a difference in steady state between the control group and the experimental group" (principlesofchaos.org), so a degraded starting point means any difference observed cannot be attributed to the injected fault: an uninterpretable result carrying full risk.

Gating on "at least 99% of instances healthy" is a practitioner convention, not a standard. Treat it as a starting default. The defensible version is stricter and local: the baseline must sit inside the range the service normally holds at this time of day, and be stable across the measurement window.

Gate 3: observability is live

Confirm that fresh samples for every abort-criterion signal have arrived within the last sampling interval. Stale dashboards fail this gate exactly as hard as missing ones.

The signals to confirm are the ones the drill will abort on. A useful default set is the four golden signals from sre.google/sre-book/monitoring-distributed-systems:

Without live signals there is no way to detect that a bound has been crossed, which means the drill has no abort path that fires on evidence. That is not a riskier drill. It is a fault injection with no stopping rule.

Gate 4: the rollback has been exercised, not documented

Verified means someone ran it and watched it succeed. A runbook entry, a tool's advertised time-to-live, or a colleague's recollection that "it worked last time" do not clear this gate.

Exercise it like this, before injection:

1. With no fault present, run the exact rollback action against the exact target, from the exact account or context that will run it during the drill.
2. Confirm it returns success and leaves the system unchanged.
3. Record who ran it, when, and how long it took. That duration becomes the assumed abort latency in the drill contract.

Where the injection tool offers an automatic expiry as well, set it, and treat it as a backstop behind the manual action rather than as the rollback itself.

This gate is the direct lesson of the SRE test-induced emergency, where the rollback path was flawed because the team had not "tested our rollback procedures in a test environment" (sre.google/sre-book/emergency-response). Untested paths behave the same everywhere: the same book warns that depending on a code path with no coverage lets an unrelated change in the environment silently change the result (sre.google/sre-book/testing-reliability).

Gate summary

Stage 2: choosing a blast-radius bound

Per principlesofchaos.org, "It is the responsibility and obligation of the Chaos Engineer to ensure the fallout from experiments are minimized and contained." That principle sets the direction but not the number. Derive the number instead of asserting it.

The load argument. If a fraction f of N replicas is removed, each surviving replica absorbs the redirected share, so its load multiplies by 1 / (1 - f). That is arithmetic, and it is the reason a large bound quietly turns into two experiments at once:

At 50% the drill is testing the injected fault and a doubling of per-replica load simultaneously, so a failure cannot be attributed to either one. At small fractions the confound is small and the result is readable. That is the whole argument for starting small and widening only after clean runs.

The reference point for "small". Netflix's Chaos Monkey, running against production, ships a deliberately tiny bound: every weekday it "flips a weighted coin to decide whether to terminate an instance from that group", and when the coin comes up heads it schedules one termination in a daytime window (netflix.github.io/chaosmonkey/Termination-behavior). One instance per group per day, on purpose, in the most mature chaos program there is. A first drill has no case for being bolder than that.

Expressing the error bound. State the error-rate ceiling against the service's remaining error budget rather than as a bare percentage. Per sre.google/sre-book/embracing-risk, the error budget is the difference between the SLO target and measured actual uptime, "the 'budget' of how much 'unreliability' is remaining". A drill spends that budget. Deciding in advance what share of it this drill may spend is what keeps a series of drills from consuming the quarter.

Duration. Bound it in wall-clock time, and make the bound short enough that the full duration plus the rehearsed rollback duration fits inside the window the team has agreed to stay attentive for. A drill that outlives its watchers has no abort path.

Stage 3: monitoring while the fault is live

Abort criteria are written before injection, never judged live

Every abort criterion is a signal, a threshold, and a dwell time, written into the drill contract before injection and unchanged once the fault starts.

This is not a formality. Once a fault is live, the person watching is under time pressure, has already invested effort in getting the drill scheduled, and is looking at a graph that is supposed to be moving. Those are the conditions under which a threshold gets renegotiated in the moment. Writing the number in advance removes the decision from that moment: the sampler compares, it does not deliberate. The scientific framing demands the same, since the experiment exists to "try to disprove the hypothesis" (principlesofchaos.org), and a hypothesis whose bounds move during the run cannot be disproved.

Nobody may loosen a criterion mid-run. Anyone may abort early. Those are not symmetric, and should not be.

Sampling cadence

Read every abort-criterion signal on a fixed interval. A 10-second interval is a practitioner convention, not a standard. Two constraints set the real value:

• Shorter than the tightest dwell time, by several samples. A criterion of "p99 above 10x baseline for 60 seconds" needs enough samples inside that 60 seconds to establish the condition held rather than spiked. Ten seconds gives six. One sample per minute gives one, which cannot distinguish a sustained breach from a single noisy point.
• No shorter than the telemetry backend's own resolution. Sampling faster than the backend aggregates returns repeated values and creates false confidence in the cadence.

Pick the interval from those two constraints for the actual stack, then write it into the contract.

What ends the drill early

Each row is filled in with the specific numbers before injection. The example values below are illustrative shapes, not recommended defaults.

The abort itself

1. Run the rehearsed rollback action. It is the one from Gate 4, unchanged.
2. Record the timestamp, the criterion that fired, and every signal value at that moment, before anything is cleaned up.
3. Proceed to Stage 4. An aborted drill still gets recovery validation.

Never abort silently. An abort is the drill's most informative outcome: it is the moment the system told you where its bound is. Discarding it because the run "did not complete" throws away the only data the drill produced.

Stage 4: recovery validation

Recovery is confirmed, not assumed. The property under test is recoverability, defined in the ISTQB glossary (V4.7.2) as "the degree to which a component or system can recover the data directly affected by an interruption or a failure and re-establish the desired state of the component or system", after ISO 25010 (glossary.istqb.org/en_US/term/recoverability).

Removing the fault is not recovery. Recovery is the system returning to the baseline recorded at Gate 2.

The checks

Tolerance and timeout

"Baseline plus 10%" is a practitioner convention, not a standard, and a poor final answer because it is unrelated to how noisy the specific signal is. The tolerance has two opposing constraints:

• Wide enough to clear. Inside the signal's own run-to-run variation, recovery never validates and the team learns to ignore the check.
• Narrow enough to mean something. Too wide, and a system left measurably degraded passes as recovered, which is the failure this stage exists to catch.

Derive it from the Gate 2 measurement: set the tolerance from the observed variation of that signal across the baseline window and write the number into the contract. Keep the 10% default only where no baseline variance was captured, and say so in the record.

Set a maximum wait too. Five minutes is a common convention; the right value depends on the system's own restart and warm-up behavior. Reaching the timeout is a result, not an error: recovery did not complete unassisted, which is a finding worth more than a clean pass.

Verdicts

Never run the next drill from an unrecovered state. A follow-up launched before recovery validates starts from an unknown baseline, which fails Gate 2 by definition, and its blast-radius reasoning is void because the surviving-capacity arithmetic assumed a full complement of healthy replicas.

Worked example

Service checkout, staging, N = 12 replicas. The experiment definition and its hypothesis already exist.

Drill contract, agreed and frozen before injection

hypothesis:
  metric: checkout_completion_rate
  threshold: ">= 95%"
  window: "5m"
target:
  environment: staging
  namespace: checkout-staging
  service: checkout
  replicas: 12
fault: pod-kill
blast_radius:
  max_replicas_affected: 3        # 25% of 12; survivors carry about 1.33x load
  max_duration: "5m"
sample_interval: "10s"            # convention; 6 samples inside the 60s dwell time
abort_criteria:
  - id: error-budget
    signal: http_5xx_rate
    threshold: "> 2%"             # agreed share of remaining quarterly error budget
    dwell: "30s"
  - id: blast-radius
    signal: unready_replicas
    threshold: "> 3"
    dwell: "0s"
  - id: latency
    signal: p99_latency
    threshold: "> 10x baseline"
    dwell: "60s"
  - id: downstream
    signal: payments_slo_state
    threshold: "breached"
    dwell: "0s"
recovery_criteria:
  ready_replicas: "== 12"
  error_rate: "<= baseline + 0.4%"   # from observed baseline variance, not the 10% default
  p99_latency: "<= baseline + 12%"   # from observed baseline variance
  settle_window: "2m"
  timeout: "5m"
rollback:
  action: "remove the injected pod-kill fault from the checkout-staging namespace"
  owner: "on-call SRE, present for the full run"
  rehearsed: "yes, run clean 8 minutes before injection, completed in 6s"

Pre-flight

All four passed, so the drill proceeds. Had any single gate failed, the drill would have stopped here.

Live run

T+0:00  inject pod-kill, 3 of 12 replicas targeted
T+0:10  ready 10/12  5xx 0.3%  p99 268ms   within all bounds
T+0:20  ready  9/12  5xx 0.6%  p99 310ms   within all bounds
T+0:30  ready  9/12  5xx 1.1%  p99 402ms   within all bounds
T+0:40  ready  8/12  5xx 2.4%  p99 620ms   5xx above 2%, dwell timer starts
T+0:50  ready  8/12  5xx 2.9%  p99 880ms   5xx still above 2%, dwell 20s
T+1:00  ready  8/12  5xx 3.3%  p99 1140ms  5xx above 2% for 30s -> ABORT
T+1:06  rollback complete (6s, matching the rehearsal)

The error-budget criterion fired at its written threshold and dwell time. No one relitigated the 2% while the graph was climbing. Note also that a fourth replica went unready at T+0:40 without being targeted, which is the cascade the blast-radius criterion exists to catch and which would have fired at 4.

Recovery validation

T+1:06  rollback complete, settle window starts
T+2:10  ready 11/12  5xx 0.9%  p99 430ms
T+3:20  ready 12/12  5xx 0.3%  p99 262ms   all checks inside tolerance
T+5:20  all checks held continuously for the 2m settle window -> RECOVERED

Verdict RECOVERED, unassisted, 4m14s after rollback, inside the 5m timeout.

Expected output shape

One record per drill, aborted runs included.

## Chaos drill record: checkout-pod-kill-2026-07-19

**Target:** staging / checkout-staging / checkout (N=12)
**Fault:** pod-kill, bound 3 of 12, max 5m
**Hypothesis:** checkout_completion_rate >= 95% over 5m
**Outcome:** ABORTED at T+1:00 | **Recovery:** RECOVERED (4m14s, unassisted)

### Pre-flight
| Gate | Result | Evidence |
|---|---|---|
| Non-production target | PASS | context checkout-staging |
| Healthy measured baseline | PASS | 12/12 ready, 5xx 0.2%, p99 240ms, stable 10m |
| Live observability | PASS | freshest sample 4s old, all 4 signals |
| Exercised rollback | PASS | rehearsed T-8m, success in 6s |

### Run
Injected T+0:00, aborted T+1:00 on `error-budget` (5xx > 2% sustained 30s),
rollback complete T+1:06. Signal values at abort: ready 8/12, 5xx 3.3%,
p99 1140ms.

### Observed blast radius
| Measure | Peak observed | Bound |
|---|---|---|
| Replicas unready | 4 of 12 | 3 of 12 |
| Error rate | 3.3% | 2% |
| p99 latency | 1140ms | 10x baseline (2400ms) |
| Downstream | payments within SLO throughout | any breach aborts |

### Recovery
Capacity 12/12 at T+3:20; errors 0.3% (inside baseline + 0.4%); p99 262ms
(inside baseline + 12%); payments within SLO; held through the 2m settle
window. Verdict `RECOVERED`.

### Findings and next step
- Hypothesis did not hold: completion rate fell below 95% before the bound.
- Cascade beyond the target: a fourth replica went unready without being
  targeted. Investigate the shared dependency before widening the bound.
- Do not widen the blast radius; re-run the same contract once understood.

Anti-patterns

Limitations

• Gates reduce risk, they do not remove it. The SRE test-induced emergency passed internal review and still took out dependent services, because the team's model of the dependencies was incomplete (sre.google/sre-book/emergency-response). A drill can always surface a coupling nobody knew about: that is the point, and the residual risk.
• Non-production is a weaker signal than production. Staging differs in traffic shape, data volume, and dependency versions, so a clean drill there is evidence, not proof. Production experiments are what principlesofchaos.org prefers, and reaching them is a separate organizational decision, not an extension of this protocol.
• The tolerances and intervals here are conventions. The 10-second sampling interval, the "baseline plus 10%" recovery tolerance, and the five-minute recovery timeout are common practice, not standards. Replace each with a value derived from the system's own measured behavior.
• Recovery validation confirms signals, not correctness. Metrics returning to baseline does not prove data written during the fault is consistent. Correctness after a fault needs its own verification.