quality-status-digest

Overview

A quality status digest is a backward-looking, one-page answer to "where does quality stand right now". It is assembled from numbers the team already produces: how many CI runs finished and how, how many defects reached production, how many tests are sitting in quarantine. The work is not measurement. The work is the denominator rules, the thresholds, and the discipline to keep unlike metrics in unlike buckets.

This skill has two altitudes that share one method. Part 1 computes a per-team digest. Part 2 rolls several per-team digests into a portfolio view. The roll-up is not a second calculation: it reads the values Part 1 already emitted (that is what the summary row in Part 1's output template is for) and never re-derives them from raw run data. Re-deriving is how two documents in the same meeting end up disagreeing about the same team.

What this skill owns, and what it does not

Owns: turning metrics that already exist into the recurring status artifact - the denominators, the score formulas, the red / amber / green cut points, the tagging scheme, and the two output templates.

Does not own:

• Instrumenting anything. If pass rate is not computable because CI does not record run conclusions, or escapes are not countable because production defects are untagged, this skill reports the gap under Open items. It does not add tracking.
• Defining SLOs, targets, or OKRs. Thresholds here classify what already happened. A commitment about what quality should be next quarter is a different artifact with a different author.
• Deciding priorities. The digest ranks risk and flags where the trend is bad. Choosing what the team works on next is a planning decision made with the digest as one input, not a line the digest writes.
• Root-cause work. The digest counts escapes and flakes. Classifying an escape as a test gap versus a process gap, or bisecting a flake to its triggering condition, is downstream investigation.

Inputs the computation assumes

You need these values for the reporting window (default seven calendar days per team; a quarter for the portfolio view). How you obtain them is out of scope: any CI API, tracker export, or manifest works.

For the portfolio view, add per team: the emitted per-team summary row, an estimate of blast radius (users or revenue exposed), current headcount, and open roles.

Part 1 - The per-team digest

Step 1 - Pass rate, with the denominator rule

pass_rate = successful_runs / (successful_runs + failed_runs)

Cancelled and skipped runs are excluded from the denominator. They carry no information about whether the software works: a cancelled run reports on a human pressing a button or a newer commit superseding an older one. Leaving them in makes pass rate move whenever CI hygiene changes, which is the single most common way this number becomes uninterpretable.

Compute the same figure for the prior window. The delta, expressed in percentage points, is the trend. Report both: a 78% pass rate that rose 9 pp and a 78% that fell 9 pp are opposite situations with the same headline.

Step 2 - Escape defects (and why this is not a DORA metric)

An escape defect is one that reached production despite the existing test suite. Count the escapes created in the window. Where deployment count is known:

escape_rate = escapes_in_window / deployments_in_window

If deployment count is unavailable, report the raw count and say the denominator is unknown, in the digest itself. Do not silently switch denominators between windows; the trend becomes meaningless.

Escape-defect rate is not a DORA metric. Neither is pass rate, nor flake debt. DORA measures software delivery performance; its current guidance documents five metrics grouped as throughput and instability, with verbatim definitions in references/dora-metrics.md.

Deployment frequency and change fail rate are usually computable from the same CI data the digest already reads, so include them as delivery context in their own section. Change lead time and failed deployment recovery time need commit timestamps and incident records, so mark them partial rather than guessing. Escape rate belongs beside these figures, never inside them: defect leakage and delivery performance answer different questions and a single informed reader will catch the conflation.

Step 3 - Flake debt

flake_debt_score = (stale_quarantine_count * 2) + new_flakes_in_window

Count an entry as stale when it has been quarantined more than 14 days. Both numbers in this formula are practitioner conventions, not standards:

• 14 days approximates one two-week sprint. An entry that survives a full planning cycle has outlived the "we will fix it next sprint" promise that justified quarantining it, which is exactly the population you want counted separately. On a one-week cadence, set it to 7.
• The weight of 2 encodes a judgement: a long-lived quarantine entry is coverage that is already gone and keeps widening, whereas a new flake is usually still fresh in someone's memory and cheap to fix. Weighting stale entries double makes a queue of old entries outrank a burst of new ones. If your team fixes old entries as readily as new ones, the weight is 1.

State the weight and the staleness cut-off in the digest, so a reader who disagrees can recompute rather than dismiss.

Step 4 - Red / amber / green per area

The cut points below are conventions, and a starting point only. They are not benchmarks and no standard sets them. Calibrate against your own baseline and record which basis you used.

The headline status for the team is the worst area, not an average. Averaging a red flake-debt score against a green pass rate hides the only line that needed a decision.

Step 5 - Emit the per-team digest

Write one file per window, for example quality-digest/<YYYY-MM-DD>.md, carrying a header (window, threshold basis, deployment definition, flake weight), a Summary table (one row per area: status, metric, trend), a headline (the worst area), a section per metric showing its inputs, a partial DORA delivery-context section, Top risks, and Open items. The final fenced digest-row line is what the portfolio roll-up consumes, so emit it even when only one team reports:

digest-row: team=checkout window=2026-07-06..2026-07-12 pass_rate=0.94 delta_pp=+2 escapes=1 deployments=12 flake_debt=12 rag=RED basis=defaults

See references/output-templates.md for the full per-team digest template.

Part 2 - The portfolio roll-up

Run this when several teams have each produced a Part 1 digest for comparable windows. It consumes their summary rows.

Step 6 - Build the cross-team table from the emitted rows

Parse one digest-row per team and place each value in a row unchanged. Do not recompute pass rate, escapes, or flake debt from raw CI at this altitude: the per-team digest already stated its window, its deployment definition, and its threshold basis, and re-deriving without those settings produces numbers that contradict the documents leadership already received.

Carry the threshold basis into the portfolio table. Two teams both showing GREEN under different calibrations are not equivalent, and the reader needs to see that.

Portfolio headline: RED if any team is RED. This is a convention, chosen because a portfolio view exists to surface the worst case; averaging four teams into AMBER describes no team that exists and hides the one needing a decision.

Add the two portable DORA figures per team, deployment frequency and change fail rate (definitions in references/dora-metrics.md). These two survive cross-team comparison best because they need only deployment records and incident flags, while lead time and recovery time depend on per-team commit and incident conventions that rarely match.

Step 7 - Risk heatmap

Score each team on two axes and place it on a 3 x 3 grid:

• Signal severity: the team's worst digest area, taken straight from its headline (GREEN / AMBER / RED).
• Blast radius: users or revenue exposed if that area fails, in three bands (low / medium / high) defined once for the whole portfolio and stated in the document.

If nobody supplied user or revenue figures, write [DATA NOT SUPPLIED] in the cell rather than estimating. An invented blast radius gets quoted back as a fact in the next meeting, and it never carries the caveat with it.

High severity plus high blast radius is the investment candidate quadrant. The grid exists so that a RED team serving an internal admin tool does not outrank an AMBER team on the checkout path.

Step 8 - Capacity flag

Per team, record QE headcount, open roles, automation ratio, and whether testers are embedded in product squads or sit in a separate function.

Flag capacity risk when open roles exceed 20% of headcount. This is a convention with no standard behind it. The reasoning: at one vacancy in five, the remaining engineers absorb both the vacant scope and the interview loops that fill it, so delivery slips for reasons no quality metric explains. Below that ratio, hiring load is usually noise. On teams of three or four, drop the percentage and use an absolute count, because 20% of three rounds to nothing meaningful.

Two industry figures are useful context when the flag prompts a staffing conversation, from the 2026 State of Testing Report (13th edition): 56.4% of respondents' teams are measured on test coverage, against 8.6% measured on business impact, and testers in cross-functional squads report roughly 27% higher pay than those in siloed QA departments (State of Testing 2026; see References). Treat these as survey context on how the field measures and structures QA work, not as a target for your own org.

Step 9 - Tag each team STABLE, WATCH, or INVEST

Compare this period's per-team digest values against the previous period's for the same three areas: pass rate, escapes, flake debt.

This three-tier scheme is a convention. The three-way split falls out of having exactly three digest areas, which gives a natural 0 / 1 / 2-or-more boundary. If your digest tracks four areas, move the INVEST boundary rather than keeping "two" out of habit.

For each INVEST team, check whether an existing commitment covers the regressing area. A team tagged INVEST with no commitment touching the regression has no funded recovery path, and surfacing that gap is the point of the tag. The digest names the gap. It does not decide the remedy or its priority.

Step 10 - Emit the portfolio review

Emit four sections plus a scope note: the per-team roll-up table (one row per team, every value copied from that team's digest-row, never recomputed, with the threshold basis carried across), the severity-by-blast-radius heatmap, the capacity table with the open/headcount flag, and an INVEST-teams list naming each regressing area and whether a commitment covers it. Close with a "What this review did not consider" note (IC performance, vendor contracts, roadmap risk, anything outside the teams' own digests). See references/output-templates.md for the full portfolio review template.

Anti-patterns

Limitations

• Tracker data quality bounds the escape count. If production defects are inconsistently labelled, or escapes are mixed with feature requests, the escape count understates reality. Record the labelling gap under Open items rather than presenting the count as complete.
• "Deployment" is not a portable word. It may mean a production release, a staging push, or a feature-flag flip. Escape rate and deployment frequency are only comparable across teams that share the definition, which is why the digest header states it.
• Flake debt is a proxy. Quarantine age and new-flake count are observable stand-ins for instability. They cannot see flakes that were never quarantined, so a team with no quarantine process can score 0 while being the least stable in the portfolio.
• DORA context is partial from CI data alone. Change lead time and failed deployment recovery time need commit timestamps and incident records (dora.dev); reporting them as computed when they were estimated is worse than omitting them.
• Trend tags need a comparable prior period. A first portfolio review has no STABLE / WATCH / INVEST tags to give, only a baseline. Say so instead of tagging everything STABLE by default.

References

• DORA software delivery metrics - five-metric model and the verbatim definitions used in Steps 2 and 6: https://dora.dev/guides/dora-metrics-four-keys/ (verified 2026-07-19). Full definitions in references/dora-metrics.md.
• 2026 State of Testing Report (13th edition) - the coverage / business-impact and cross-functional-pay figures cited in Step 8: https://www.practitest.com/state-of-testing/ (verified 2026-07-19).