Anti-patterns

Anti-patterns

Output template

Produce one document. It is the plan before the window and the record after it.

# Release cutover plan - {release_name}

**Window:** {start_utc} to {hard_stop_utc}
**Release authority:** {one named person}
**Hard stop consequence:** reaching {hard_stop_utc} with open gates puts the
full reverse path to the release authority as a decision.

## Rollback rule for this window

Rollback is a decision made by {release authority} on stated evidence.
Thresholds halt the sequence; they do not reverse it. Recovery may be
roll back, roll forward, or redeploy last-known-good, and only the release
authority chooses which.

## Dependency graph

{service -> service edges, one per line}
{parallel tracks listed explicitly}
{preconditions on out-of-window services, each with the person who confirmed it}

## Gate sequence

| Gate | Kind | Step | Depends on | Owner | Timebox (UTC) | Status |
|------|------|------|-----------|-------|---------------|--------|

## Authority table

| Role | Named person | Scope |
|------|--------------|-------|
| Release authority | | all DECISION gates and recovery calls |
| {service} owner | | {gate IDs} execution |

## Rollback triggers

| Gate | Condition that puts a decision on the table | Evaluated by | Evidence | Reverse scope |
|------|--------------------------------------------|--------------|----------|---------------|

## Reverse path

{completed ACTION gates in reverse order, one owner per step,
 confirmation required between steps}
{gates marked POINT OF NO RETURN and what recovery means past them}

## Runtime log

| Time (UTC) | Gate | Action | Verdict | Evidence | Called by |
|------------|------|--------|---------|----------|-----------|

Update the Status column and append to the runtime log in place as the window runs. Authority handoffs are log rows. At close, the document is the release record, and the coupling that forced a coordinated window is worth carrying into the retrospective.

Cutover sequencing - multi-team release windows

Deep dive for release-runbook-author. Sequences a multi-team release cutover into dependency-ordered gates: the cross-service dependency graph, a numbered gate list where every gate carries exactly one named owner, a hard timebox, and a written rollback trigger, then the reverse-order rollback path and the window hard-stop rule. Consult when two or more teams must cut over interdependent services inside one shared release window and nobody has yet written down the order, who calls each gate, or what reverses it.

What this owns, and what it does not

This owns the cross-team sequencing problem: several services, owned by several teams, that must change over in a specific order inside one shared window, where the failure mode is "team C started before team A's gate was confirmed" and "nobody knew who was allowed to say stop".

It deliberately does not cover the single-service release runbook: the pre-flight checklist, the smoke gate, canary observation thresholds and their statistical comparison, the human promote gate, the progressive rollout, and post-release verification for one service. That is the host release-runbook-author SKILL.md, run per service, inside the timebox this plan gives that service. Write it separately and reference it from the gate row. If your window contains one service, you do not need this reference at all.

Needing a cross-team cutover sequence is itself a coupling signal worth recording. DORA sets the opposite target - teams releasing "independently of the services it depends on" - and names this failure as a "big-bang" deployment that forces orchestration across many hand-offs and dependencies (DORA, loosely coupled teams (opens in new window)). Sequence the window you have, and put the coupling that forced it into the post-window record.

The rule that governs every gate

Rollback is a named human decision made on evidence. It is never an automatic metric trigger.

A threshold being crossed is an input to that decision, not the decision. Published guidance lists three distinct, non-interchangeable recoveries from a bad deployment:

• Rolling back - undo the changes and revert to the last known working configuration.
• Rolling forward - address the issue mid-rollout with a hotfix.
• Deploying new infrastructure - stand up the last known working configuration afresh (Azure Well-Architected, safe deployment practices (opens in new window)).

Something has to choose between those, and in a multi-team window that choice also determines how many other teams reverse, so it cannot be delegated to a threshold in one service's monitoring.

What is automatic is the halt, not the reversal. When a health signal trips during a rollout phase, "the rollout should immediately halt" and an investigation into the alert determines the next course of action (Azure Well-Architected, safe deployment practices (opens in new window)). So: automation stops the sequence, a named human restarts it or reverses it.

Every gate table below therefore has an owner column and a trigger column, and the trigger text is always a condition that puts a decision in front of a person, never an action.

Step 1 - Build the dependency graph

For each service in the window, record three things:

Then write edges in one direction only: "X must be live before Y". If Y calls a contract that only exists in X's new version, X cuts over first. If neither consumes the other's changed surface, there is no edge and they are parallel tracks.

Three graph shapes need handling before you can sequence anything:

• A cycle. X needs Y's new version and Y needs X's new version. This cannot be ordered. Break it before the window by making one side accept both versions, or by shipping the new path behind a flag: feature flags "can help you control the exposure of new code and quickly roll back deployment if issues arise" (Azure Well-Architected, safe deployment practices (opens in new window)). A window whose graph has a cycle is not ready to be scheduled.
• A shared schema. Two services on one database are coupled even with no API edge. The blue-green write-up handles this by applying the database refactoring first, so the schema supports both the old and the new application version, verifying that, and only then deploying the new code (Martin Fowler, BlueGreenDeployment (opens in new window)). Do the schema step before the window, as its own change, not as a gate inside it.
• An unowned edge. A dependency whose upstream service is not in the window and not owned by a participating team. It is a precondition, not a gate. List it as a precondition with a named person who confirmed it, and move on.

Record the graph as text, not as a picture, because the gate list is generated from it and the runtime log has to quote it.

Step 2 - Convert the graph into an ordered gate list

Walk the graph in dependency order. Each service contributes at least two gates, and the distinction between them is the whole mechanism:

Rules for generating the list:

1. Never merge an ACTION and a DECISION into one row. The row that flips the router cannot also be the row that judges whether the flip worked.
2. Every ACTION gate that sits on a dependency edge is followed by a DECISION gate before any dependent service's ACTION gate may start. This is the cross-team invariant. Health checks gating each phase is the published shape: "Deployments must pass health checks before each phase of progressive exposure can begin" (Azure Well-Architected, safe deployment practices (opens in new window)).
3. Parallel tracks get their own gate IDs and their own DECISION gates. They do not borrow another track's confirmation.
4. Number gates in execution order (G0, G1, G2, ...) but keep a Depends on column carrying the real graph. The numbering is for talking on a call. The Depends on column is what is actually true, and it is what the reverse path in Step 6 is derived from.
5. Expand outward from the smallest blast radius. The phased shape is standard practice: fit the deployment process to the risk profile of the service, and "push by starting in one cluster and expand exponentially until all clusters are updated" (Google SRE Book, Release Engineering (opens in new window)). In a cross-team window each service's own expansion happens inside its timebox, under its own single-service runbook. The cross-team plan sequences the services, not the shards.

The governing principle for the whole list, from the same chapter: "Changes to any aspect of the release process should be intentional, rather than accidental" (Google SRE Book, Release Engineering (opens in new window)). A gate that exists because the sequence needs it, with no owner and no evidence, is an accident waiting to be discovered at 02:00.

Step 3 - Assign exactly one named owner per gate

One human name per gate. Not a team, not a rota alias, not a Slack channel.

• ACTION gates belong to the owning team. Teams run their own releases: release engineering practice "allow our product development teams to control and run their own release processes" (Google SRE Book, Release Engineering (opens in new window)). The cross-team plan does not tell a team how to deploy its service.
• Every DECISION gate belongs to one release authority for the whole window. This borrows the incident command model deliberately, and it is worth being explicit that the source is incident response rather than planned releases: the incident commander "holds the high-level state about the incident" and "structure[s] the incident response task force, assigning responsibilities according to need and priority", and the reason for the single role is that "it's important to make sure that everybody involved in the incident knows their role and doesn't stray onto someone else's turf" (Google SRE Book, Managing Incidents (opens in new window)). A cutover window has the same property: many teams acting concurrently on shared production state, needing one place where authority sits.
• Handoff is explicit and acknowledged. If the window crosses a shift boundary, the authority transfers by statement, not by drift. The incident handoff protocol is the model: the outgoing commander "should be explicit in their handoff, specifically stating, 'You're now the incident commander, okay?', and should not leave the call until receiving firm acknowledgment of handoff" (Google SRE Book, Managing Incidents (opens in new window)). Record the handoff as a row in the runtime log with both names and the clock time.

Write the assignment as its own table so nobody has to reconstruct it from the gate list mid-window.

Step 4 - Set the timebox and the hard stop

A timebox is the wall-clock time by which the gate must be cleared, not an estimate of how long the work takes. Estimates slip quietly; deadlines are observable.

Say this plainly to whoever reads the plan: the specific clock values are a scheduling convention, not a published standard. No source cited here prescribes "10 minutes for a router switch". What published guidance does constrain is any gate with an observation period: "Bake times should be measured in hours and days rather than minutes" and should increase per rollout group to cover different time zones and usage patterns (Azure Well-Architected, safe deployment practices (opens in new window)).

That has a sharp consequence for cross-team windows, and it is the most common thing this plan gets wrong: a gate labelled "observe for 10 minutes" is not a bake period. It is a smoke check. If a service genuinely requires a bake before its dependents proceed, the dependents do not belong in the same window. Either split the window across days, or accept in writing that the window is buying smoke coverage rather than bake coverage. Do not relabel one as the other.

To size the window:

1. Sum the ACTION durations along the critical path, the longest chain in the Depends on graph. Parallel tracks do not add to it.
2. Add each DECISION gate's evidence-gathering time on that path.
3. Add slack, conventionally 20 to 30 percent of the total. This figure is a practitioner convention, not a published standard.
4. Compare against the window length. If it does not fit, cut a service out of the window. Do not shrink observation time to make the arithmetic work.

The hard stop is one wall-clock time for the whole window, set once, in the plan. Reaching it with gates incomplete triggers the reverse-order rollback path from Step 6. Extending past it is allowed, but only as an explicit DECISION gate called by the same named release authority, recorded in the log with a reason. Emergency acceleration follows the same rule: define in advance "who can approve SDP acceleration in an emergency and the criteria that must be met for acceleration to be approved" (Azure Well-Architected, safe deployment practices (opens in new window)).

Step 5 - Write the rollback trigger for each gate

Every gate gets a trigger, written before the window, with three parts:

Trigger classes worth covering per service:

Two things the trigger text must not do:

• It must not name an action. "Roll back if error rate above 1 percent" is wrong. "Error rate above 1 percent puts a recovery decision to the release authority" is right. The reason is the governing rule stated above: rollback, roll forward, and redeploy-known-good are three different recoveries, and a threshold cannot pick among them (Azure Well-Architected, safe deployment practices (opens in new window)).
• It must not be a feeling. "If it looks unhealthy" is not a trigger. Someone at 02:00 has to evaluate it identically to how you would.

Rollback itself, when chosen, is a planned reversible action rather than an improvisation. That is the whole point of holding the previous version ready: "if anything goes wrong you switch the router back to your blue environment" (Martin Fowler, BlueGreenDeployment (opens in new window)), and for a gradual rollout "the rollback strategy is simply to reroute users back to the old version until you have fixed the problem" (Martin Fowler, CanaryRelease (opens in new window)).

Step 6 - Derive the reverse-order rollback path

Rollback runs the dependency order backwards. The service that cut over last reverses first.

The reason is mechanical, not stylistic. If a dependency reverses while its dependent is still on the new version, the dependent is now calling a contract that no longer exists, and you have converted a bad release into an outage. So reversal is the completed prefix of the gate list, read bottom-up.

To generate it:

1. Take the gate list. Cut it at the gate where the decision is being made.
2. Reverse the completed ACTION gates.
3. Keep parallel tracks independent: a track with no edge to the failing service reverses only if the decision says the whole window reverses.
4. Assign each reverse step an owner. Default to the person who executed the forward action, because they have the context and the access.
5. Give each reverse step its own confirmation before the next one starts. A rollback sequence with no confirmations is a second uncontrolled cutover.

Mark the irreversible gates. Some forward actions do not reverse mechanically, and the plan is dishonest if it implies they do. Stateful changes are the usual case: "Rolling back changes, especially database, schema, or other stateful component changes, can be complex" (Azure Well-Architected, safe deployment practices (opens in new window)). For any gate whose forward action writes state in a shape the old version cannot read:

• Label the row POINT OF NO RETURN in the gate table.
• State in the plan that once this gate passes, recovery for that service means roll forward, not reverse.
• Prefer to have removed the problem before the window, using the schema-supports-both-versions approach from Step 1 (Martin Fowler, BlueGreenDeployment (opens in new window)).

Step 7 - Check the plan before the window opens

Run the plan against these invariants. Every one is a defect in the plan, not a judgment call:

Worked example

A four-service, three-team window (one dependency chain plus one parallel track) worked end to end - dependency graph, gate list, rollback triggers, and the reverse path: cutover-worked-example.md (opens in new window).

Output template

One document that is the plan before the window and the record after it - plan header, rollback rule, dependency graph, gate sequence, authority table, rollback triggers, reverse path, and runtime log: cutover-output-template.md (opens in new window).

Anti-patterns

Nine cutover anti-patterns with why each fails and its fix: cutover-anti-patterns.md (opens in new window).

Limitations

• No execution. This produces the plan and the record. Flipping routers, running smoke suites, and reading dashboards belong to the owning teams and their own tooling.
• Single-service depth is out of scope. Pre-flight, canary thresholds, statistical promote criteria, and post-release verification for one service live in that service's own runbook, run inside its timebox here.
• Outcomes are reported, not polled. The runtime log records what owners state, with the evidence they cite. It is not a monitoring integration.
• Cross-timezone availability is flagged, not solved. The Step 7 check surfaces owners who have not confirmed availability for their gate's clock time. Resolving that is a scheduling conversation.
• Shared-database sequencing is a precondition, not a gate. Schema work that must support two application versions simultaneously is resolved with the data owner before the window opens.

Worked example

Window: four services, three teams, one dependency chain plus one parallel track.

Graph. The payments API calls the new authentication contract, so authentication must be live first. The web app calls the new payments contract, so payments must be live before it. The event pipeline consumes only published events whose shape is unchanged, so it has no in-window edge and runs parallel.

auth-service  ->  payment-api  ->  web-app
event-pipeline    (parallel, no in-window dependency)

Gate list.

Hard stop: 22:00 UTC. Reaching it with any gate incomplete puts the full reverse path to Priya as a decision.

Rollback triggers.

Reverse path if the decision at G8 is to roll back the window. Completed ACTION gates are G1, G3, G5, G7. Reversed, with the parallel track handled independently:

1. web-app router back to previous version      (Carol)  confirm
2. payment-api router back to previous version  (Bob)    confirm
3. auth-service router back to previous version (Alice)  confirm
4. event-pipeline router back (parallel track)  (Dave)   confirm

Each step confirmed before the next begins. Note that G3's forward action was marked reversible during Step 6 review; had it written transaction rows in a new shape, it would carry POINT OF NO RETURN and step 2 above would read "roll forward with hotfix" instead.

Output template

One document. It is the plan before the release and the record after it.

# Release runbook - {service} {version}

**Promote-gate owner:** {one named person, availability window}
**Last known good artifact:** {id}
**Recovery rule:** a crossed threshold halts the phase and puts a recovery
decision to {owner}. Recovery may be roll back, roll forward, or redeploy
last known good, and only {owner} chooses which.

## Baseline - recorded {window} before deploy

| Metric | Value | Query |
|--------|-------|-------|

## Thresholds

| Metric | Absolute floor | Ratio limit (canary vs control) | Ratio limit (rollout vs baseline) |
|--------|----------------|----------------------------------|------------------------------------|

## Phase 1 - Pre-flight

| Check | Verdict | Evidence |
|-------|---------|----------|

## Phase 2 - Smoke gate

**Command:** **Environment:** **Duration:** **Result:**

## Phase 3 - Canary  ({share} traffic, {window}, coverage: smoke | bake)

| Metric | Absolute floor | Ratio limit | Control | Canary | Ratio | Verdict |
|--------|----------------|-------------|---------|--------|-------|---------|

**Anomalies below threshold:**
**Verdict:** PASS | PROCEED WITH CAUTION | HALT

## Phase 4 - Promote gate

**Decision:** continue | pause | rollback
**Made by:** **At:** **On this evidence:**
**Acknowledged anomalies carried forward:**

## Phase 5 - Rollout

| Stage | Share | Window | Metrics vs baseline | Verdict |
|-------|-------|--------|---------------------|---------|

## Phase 6 - Post-release

| Metric | Baseline | Observed | Ratio | Verdict |
|--------|----------|----------|-------|---------|

**Administrative tail:** tag / changelog / notification, each with a timestamp.

## Follow-ups

- [ ] {product defects acknowledged at the promote gate}
- [ ] {runbook defects the release exposed}

Worked example

Service checkout-api, release v1.4.5, single service, no cross-team dependencies.

Baseline recorded at 13:55 UTC, 60-minute window: 5xx rate 0.31 percent, p95 latency 240ms, checkout completion 92.1 percent, distinct error signatures 41.

Phase 1, pre-flight.

Phase 2, smoke gate. npm run smoke -- --target=staging, 4m32s, 22 tests, 0 failures. PASS.

Phase 3, canary. 5 percent traffic from 14:33, 30-minute window, labelled in the runbook as smoke coverage rather than a bake. Table as shown in phase 3 above: all four thresholds green, two new error signatures observed.

NullPointerException at Cart.addItem:42   1 occurrence, absent from control
RateLimitExceeded                          1 occurrence, present in control at similar rate

Verdict: PROCEED WITH CAUTION. The rate-limit signature also appears in the control, so it is not attributable to the change. The null-pointer signature is absent from the control population and is therefore a real delta, below every threshold.

Phase 4, promote gate. Priya, 15:05 UTC, chose continue, on the evidence of the canary table plus the two signatures. The null pointer became follow-up item 1.

Phase 5, rollout. 25 percent at 15:08, observed 20 minutes, ratios within widened limits. 100 percent at 15:30.

Phase 6, post-release. 60-minute window to 16:30 against the 13:55 baseline: 5xx 0.33 percent (1.06x), p95 244ms (1.02x), checkout completion 92.0 percent (1.00x). Stable. Tag, changelog, and notification issued at 16:32. Follow-ups: the null pointer, plus a runbook defect - a 30-minute canary was too short to accumulate enough occurrences of the new signature for the promote gate to judge it confidently, so the canary window for this service moves to 60 minutes.