Risk-matrix calibration

Deep reference for the risk-matrix SKILL.md. Checks an already-written matrix against what actually happened: maps each row's likelihood rating to observed defect density, test failure rate and code churn, maps its impact rating to the severity mix and escape rate, then classifies each row as over-stated, under-stated, in-agreement, or not calibrated. Every proposed rating change carries the observation that produced it, and every proposal is handed to the matrix owner rather than applied.

Use when a matrix has been driving test decisions for at least three releases and nobody has yet checked whether its ratings match the defects, escapes and incidents that followed.

What calibration answers

A risk matrix records what a team believed, at authoring time, about how likely each area is to fail and how bad that failure would be. Risk-based testing is defined by ISTQB as "a test approach in which the management, selection, prioritization, and use of test activities and resources are based on corresponding risk types and risk levels" (https://glossary.istqb.org/en_US/term/risk-based-testing), so those ratings directly control where test effort goes. Lightweight risk assessment scores a row on two factors, likelihood and impact (https://en.wikipedia.org/wiki/Risk-based_testing).

Calibration asks one question: over the window that has since elapsed, did the things the matrix said were likely and severe actually turn out to be likely and severe? The output is a set of proposed rating changes, each attached to the observation that produced it, for the matrix owner to accept or reject.

Scope boundary

This is the after-the-fact check, not the authoring pass. The axis is already-written ratings versus observed outcomes.

If the matrix does not yet exist, or its rows have no consistent feature column and no consistent score format, there is nothing here to calibrate against. Author or restructure it first, per the SKILL.md spine.

Calibration is also not prediction. It reports what was observed in a closed historical window. It does not forecast next quarter's defects, and it does not fit a model. That restraint is deliberate: only 19.9% of teams surveyed in the PractiTest State of Testing 2026 report use AI for risk identification, against 70% for test case creation (https://www.practitest.com/state-of-testing/), and unsourced predictive scoring at the decision layer is exactly what that gap is made of.

Before anything else: what defect data can and cannot tell you

Observed defect data is a lagging and biased signal, and calibration is only as good as that signal. Read this section before running any of the steps below, because it changes what the numbers are allowed to conclude.

You only see defects in code that was tested and used. A row with zero defects in the window has at least three possible explanations:

1. The area is genuinely sound.
2. The area has no meaningful test coverage, so nothing was found.
3. The area is barely exercised in production, so nothing was reported.

No amount of defect data distinguishes these three. Absence of evidence in a defect tracker is not evidence of low likelihood. The practical rule that follows is in Step 3: a zero-defect row is never grounds for lowering a likelihood rating on its own. It is grounds for asking which of the three explanations applies, and reporting the answer.

Two further biases are worth naming explicitly:

• The tracker under-counts by construction. Escape rate is measured against defects you eventually found. ISTQB defines defect detection percentage as "the number of defects found by a test level, divided by the number found by that test level and any other means afterwards" (https://glossary.istqb.org/en_US/term/defect-detection-percentage). Defects never found by any means appear in neither the numerator nor the denominator, so a flattering escape rate can mean good testing or can mean a blind spot nobody has walked into yet.
• Labelled bug-fix data is systematically skewed. Only a fraction of bug fixes are labelled in version histories, and the resulting datasets carry "strong evidence of systematic bias" (Bird et al., ESEC/FSE 2009; full citation in the Empirical basis section below). If your team labels defects by component inconsistently, the calibration inherits that inconsistency.

State these limits in the report itself, not only here. A reader who does not know the data is biased will read a proposed downgrade as a fact.

Step 1 - Map each matrix dimension to observed signals

Each of the two rating dimensions gets its own set of observed signals. Use more than one signal per dimension so that a single weak measurement cannot move a rating on its own.

Two rules on density and churn

• Density, not raw count. A raw count conflates "small busy area" with "large stable area". Normalise by size per the ISTQB definition above; where LOC is a poor proxy, normalise by number of changes and say which denominator you used.
• Churn corroborates, never leads. Only relative (size- and window-normalised) churn predicts defect density, and only as a correlation on one commercial system; cross-project transfer of any coefficient fails even within the same domain and process. Use churn only to corroborate a direction defect density already suggested, and import no published coefficient.

The published results behind both rules (Nagappan and Ball; Zimmermann et al.), with their caveats, are in the Empirical basis section below.

Step 2 - Convert observations into an observed rating

Produce an observed likelihood and an observed impact on the same 1 to 5 scale the matrix already uses. Do not invent a new scale, and do not normalise across teams: severity labels mean different things in different trackers, so a calibration is valid only inside the conventions of the tracker it read.

The team's own authoring rubric supplies the cut points. If the matrix says likelihood 4 means "expected to fail in most releases", express the observed density in those terms rather than in a generic band. Where the matrix has no written rubric, say so in the report and mark every proposed change as provisional, because without a rubric the comparison is between one person's judgement and another's.

Record, for each row, all of:

• observed likelihood and the signals behind it,
• observed impact and the signals behind it,
• the denominator used for density,
• the count of defects the row is based on.

That last item is the honesty control. A row whose observed rating rests on two defects is a rating with a wide error bar, and the report should show the count so a reader can discount it.

Step 3 - Decide whether a divergence is worth reporting

Most rows will differ from their observations by a little. Reporting those differences buries the ones that matter and trains the owner to skim the report.

Reporting thresholds

These are practitioner conventions, not standards. No published standard sets them. The reasoning for each cut is given so a team can move it with its eyes open.

Rows that clear a threshold fall into one of three classifications.

3.1 Over-stated

The matrix rates the row higher than the observations support.

Report it as over-stated only after ruling out the three explanations in the biased-data section above. Specifically:

• If the row has near-zero defects and near-zero test executions covering it, that is a coverage question, not a low-risk finding. Report it under 3.3 instead.
• If the row has near-zero defects and near-zero production usage, that is an exposure question. The risk may be entirely real and simply not yet triggered. Say so; do not propose a downgrade.
• If the row is covered, exercised, and still quiet, an over-stated finding is reasonable. Even then, propose it as a question: a high rating may be a deliberate precaution that is working, and lowering it can remove the very testing that kept the row quiet. That circularity cannot be resolved from the data and belongs in the report as an open point.

3.2 Under-stated

The matrix rates the row lower than the observations support. This is the finding calibration exists to produce, and it is the one least vulnerable to the absence-of-evidence problem: defects that happened, happened.

Strengthen it with corroboration. An under-stated finding is strongest when defect density, test failure rate and escape rate all point the same way, and weakest when only churn does.

3.3 Coverage gap

An area that the defect data keeps naming has no row in the matrix at all, or has a row whose observations cannot be computed.

Two sub-cases, and they get different wording:

• Missing row. The area produces defects but is not represented. Handle it in Step 5 as a candidate new entry.
• Unmeasurable row. The row exists but its paths cannot be resolved, its defects are unlabelled, or no test covers it. Report the row as not calibrated and name the missing input. Do not silently score it as agreeing with the matrix, and do not score it as low risk. An unmeasurable row is a gap in the calibration, not a clean bill of health.

Step 4 - Every proposed change cites its observation

A rating change is a proposal for a human to accept, never an automatic edit. The matrix is a team artifact, usually version controlled and usually owned by a named person. Calibration produces a diff for review, and stops there.

The rule that makes review possible: every dimension of every proposed change names the observation behind it. Not a summary of the observation, a citation of it, precise enough that the owner can re-run the same query and see the same number.

A citation is adequate when it carries all four of:

1. The measured value, with its units and denominator.
2. The source it came from, identified specifically enough to re-query (which export, which filter, which paths, which time range).
3. The window the value covers.
4. The count of underlying events, so the reader can judge the error bar.

A proposed change missing any of the four is not reportable. A reviewer who cannot check the number is being asked to trust a black box, and a black box that edits the team's risk ratings is worse than no calibration at all.

Step 5 - Candidate new entries

Areas absent from the matrix that the defect data insists on are output as candidates, listed with their supporting observations and a suggested starting rating. They are never added automatically. The matrix owner decides whether the area deserves its own row or belongs inside an existing one, and that decision is about how the team models its own system, not about the data.

Two entry conditions, both practitioner conventions and both self-calibrating to the team's own data rather than to an absolute number:

• The area produced more defects in the window than the median matrix row, or
• the area produced at least one escaped defect that reached production.

Self-calibrating conditions are preferred here because an absolute cut (say, "5 or more defects") means something different for a two-person project and a two-hundred-person one.

For each candidate, propose a starting likelihood and impact, and say plainly which half of the proposal is weaker. Impact is usually the weaker half: defect data shows what broke, not what the breakage cost the user, and that translation is a product judgement.

Empirical basis for the signals

The published results behind the "churn corroborates only" and "no imported coefficient" rules, with the caveats that keep each result honest.

Why density and not raw count

A raw defect count conflates "small area, few defects" with "stable area, few defects". Normalising by size, per the ISTQB defect-density definition (opens in new window), makes rows comparable to each other. Where lines of code are a poor size proxy (heavy generated code, config-driven modules), normalise by number of changes to the same paths instead, and say in the report which denominator you used.

How much weight churn deserves

Churn is a corroborating signal, never the primary evidence for a rating change. The published relationship is real but narrow: Nagappan and Ball showed that "while absolute measures of code churn are poor predictors of defect density, our set of relative measures of code churn is highly predictive of defect density", with a metric suite that discriminated fault-prone from non-fault-prone binaries "with an accuracy of 89.0 percent" (ICSE 2005, https://www.microsoft.com/en-us/research/publication/use-of-relative-code-churn-measures-to-predict-system-defect-density/, record at https://openalex.org/W2100945416). Three limits on that result matter:

• It is a correlation, not a causal claim. Churn is a proxy for where work is happening, and work happens in areas that are being fixed, extended and reworked. High churn does not make code defective.
• The relative measures carried the result. Absolute churn, meaning a raw count of changed lines or commits, was explicitly reported as a poor predictor. Normalise churn by component size and by the length of the window before using it at all.
• The result is a case study on one commercial system (Windows Server 2003). Defect-prediction models built on one project transfer badly to another: Zimmermann et al. ran 622 cross-project predictions across 12 real-world applications and concluded that "simply using models from projects in the same domain or with the same process does not lead to accurate predictions" (ESEC/FSE 2009, DOI 10.1145/1595696.1595713, abstract at https://api.openalex.org/works/doi:10.1145/1595696.1595713, record at https://openalex.org/W25935857). Team and domain are confounds, so no published coefficient can be imported into your matrix. Use churn only to corroborate a direction that defect density already suggested.

Why labelled bug-fix data is biased

Bird et al., examining bug-fix datasets across several projects, found that only a fraction of bug fixes are labelled in version histories and reported "strong evidence of systematic bias" in the resulting datasets, which threatens both prediction models built on them and hypotheses tested with them (ESEC/FSE 2009, https://www.cs.ucdavis.edu/~filkov/papers/biasbusters.pdf, record at https://www.microsoft.com/en-us/research/publication/fair-and-balanced-bias-in-bug-fix-datasets/). If your team labels defects by component inconsistently, the calibration inherits that inconsistency.

Worked example

The expected output shape end to end: a summary, under-stated and over-stated findings with every dimension citing value / source / window / event count, not-calibrated rows, candidate new entries, and the "what was not done" footer. Names and numbers are illustrative.

# Risk-matrix calibration - checkout matrix
Window: 2026-02-01 to 2026-04-30 (3 releases). Density denominator: defects per 1k LOC.

## How to read this
Proposals only. Nothing in the matrix has been changed. Defect data is lagging
and biased: a quiet row may be sound, untested, or unused, and this report
cannot tell those apart. Rows marked "not calibrated" are gaps in the
evidence, not low risk.

## Summary
| Outcome | Rows |
|---|---|
| In agreement (below threshold) | 14 |
| Under-stated | 4 |
| Over-stated | 2 |
| Not calibrated (unmeasurable) | 3 |
| Candidate new entries | 1 |

## Under-stated

### R-12 inventory-cache: propose likelihood 2 -> 4, impact 3 -> 4 (score 6 -> 16)

| Dimension | Matrix | Observed | Observation cited |
|---|---|---|---|
| Likelihood | 2 | 4 | 13 defects over 8.4k LOC = 1.55 per 1k LOC, against a matrix-wide median of 0.31. Source: tracker export 2026-Q2, component = inventory-cache. Window: full. Events: 13. |
| Impact | 3 | 4 | 6 of 13 at severity S1 or S2 (46%); 4 of 13 escaped to production (31%), against a matrix-wide escape rate of 11%. Source: same export, field found_in. Events: 13. |
| Test failure rate (corroborating) | n/a | 6% | Suite covering services/inventory/cache fell from 99% to 94% pass rate across the window. Source: CI results 2026-Q2. Events: 412 runs. |
| Churn (corroborating only) | n/a | high | 47 commits to services/inventory/cache over 89 days, normalised 5.6 commits per 1k LOC, top decile of the repo. Corroborating only, not causal. |

All four citations carry value, source, window and event count. 13 events is
above the 5-event floor, so this is reported as a rate, not as directional.

## Over-stated

### R-03 payments-provider-fallback: propose likelihood 4 -> 2, impact unchanged at 5 (score 20 -> 10)

| Dimension | Matrix | Observed | Observation cited |
|---|---|---|---|
| Likelihood | 4 | 2 | 1 defect over 6.1k LOC = 0.16 per 1k LOC. Source: tracker export 2026-Q2, component = payments-fallback. Events: 1. |
| Impact | 5 | 5 | Single defect at S1. Too few events to move impact. Events: 1. |

**Event count is 1.** Directional only, below the 5-event floor.
Ruled out before proposing: the row is covered (214 test executions in the
window) and exercised (fallback path invoked 1,190 times in production), so
this is not a coverage or exposure artefact. **Open point for the owner:** the
high rating may be a precaution that is currently working, and lowering it
would reduce the testing that keeps it quiet. This report cannot settle that.

## Not calibrated

| Row | Missing input |
|---|---|
| R-08 legacy-tax-import | No source paths recorded; defects cannot be attributed. |
| R-15 partner-sftp-drop | Tracker component field empty on 22 of 24 defects. |
| R-21 admin-audit-log | No automated tests cover the row; failure-rate signal unavailable. |

These are evidence gaps. Do not read them as low risk.

## Candidate new entries

### notifications-webhook-retry (no row in matrix)

| Signal | Value | Observation cited |
|---|---|---|
| Defects | 8 | Tracker export 2026-Q2, component = notifications-webhook. Window: full. |
| Density | 0.94 per 1k LOC | 8 defects over 8.5k LOC; matrix-wide median 0.31. Entry condition met: above median row. |
| Escapes | 2 of 8 (25%) | Field found_in = production. Entry condition met: at least one escape. |
| Severity mix | 3 S1, 2 S2, 3 S3 | Same export. |

Suggested starting rating: likelihood 3, impact 4. **The impact half is the
weaker one**: escape rate is high, but what a missed webhook costs depends on
subscriber retry behaviour, which is a product judgement this report cannot make.

## What was not done
No matrix row was edited. No test selection was re-run. No forecast was made.

Anti-patterns

Limitations

• Defect-data quality is the ceiling. Trackers without a component or module field, or without a found-in field distinguishing test from production, cannot support an observed impact rating at all. Those rows come out as "not calibrated" and that is the correct answer.
• Path-to-row attribution is heuristic. Rows are matched to source paths, and a refactor that moved code between modules mid-window silently misattributes both defects and churn. A code-ownership file or feature registry improves this but does not fix it.
• The comparison is judgement against judgement. The observed rating still depends on the team's severity taxonomy and its authoring rubric. Where the rubric is unwritten, calibration compares one person's scale to another's and should be labelled provisional.
• Quiet rows are ambiguous by construction. This is restated here because it is the single most common way a calibration goes wrong: the report can never conclude "this area is safe", only "this area produced no evidence".
• Small windows produce unstable findings. A row flagged in one quarter and not the next may have changed, or may have had two incidents land inside one window boundary. Findings that repeat across consecutive calibrations are worth far more than a single-window flag; track which rows repeat.
• No causal claim is available. Every relationship used here is correlational, measured on someone else's system, and confounded by team and domain (https://openalex.org/W25935857). Calibration surfaces disagreement between belief and record. It does not explain the disagreement.

Heavyweight risk scoring: FMEA and Cost of Exposure

Deep reference for the risk-matrix SKILL.md. Use when the default lightweight impact-by-likelihood matrix (Step 1) is insufficient - regulated industries, safety-critical systems, or when test budget must be justified in financial terms to non-technical stakeholders.

Both methods are the heavyweight branch of risk-based testing per Risk-based testing (Wikipedia) (opens in new window), which names FMEA, Cost of Exposure, QFD, and FTA as the quantitative alternatives to lightweight scoring and describes them as tools for "financial impact analysis". They need specialized expertise; reach for them only when regulation or a money-based mitigate-or-accept decision requires it.

FMEA (Failure Mode and Effect Analysis)

Per row: failure mode + effect + severity (1-10) + occurrence (1-10) + detectability (1-10) → RPN (Risk Priority Number = S × O × D, range 1-1000).

| ID  | Failure mode                  | Effect                            | S  | O | D | RPN | Action |
|-----|--------------------------------|-----------------------------------|----|---|---|-----|--------|
| F-1 | Promo math wrong               | Wrong charge to customer           | 8  | 5 | 4 | 160 | property tests |
| F-2 | Stripe webhook missed          | Order not fulfilled                | 9  | 4 | 6 | 216 | retry + DLQ + monitor |

Detectability is the inverse of "tests would catch it" - high D = hard to catch = high RPN. Two rows with equal severity and occurrence rank differently purely on how visible the failure is to existing checks.

Cost of Exposure

Quantify the annual financial risk and weigh it against the one-time plus recurring mitigation cost:

Risk: Stripe webhook delivery failure
- Estimated incidents per year (untreated): 12
- Average revenue lost per incident: $5,000
- Annual cost of exposure: $60,000
- Cost of mitigation (retry + DLQ + monitor): $15,000 one-time + $2,000/year
- Decision: Mitigate (ROI in 3 months)

Cost of Exposure turns "this is risky" into a budget line a finance stakeholder can approve or reject.

When each applies

Product risk register

Deep reference for the risk-matrix SKILL.md. The product-level register variant: long-lived product-quality risks (functionality, performance, security, usability, compatibility, reliability) that persist across releases, distinct from the per-release matrix the spine builds. Both feed risk-based test selection and planning.

Per ISTQB CTAL-TM syllabus chapter 5 on risk-based testing and ISO 31000:2018 (risk management) - cite by stable ID; ISO behind paywall.

When to build one

• Onboarding a new test lead - establish the product-quality risk baseline.
• Quarterly product-quality review - re-score persistent risks.
• Compliance audit - document long-lived product risks beyond the current release.
• Seeding a new release matrix - copy persistent risks forward, add release-specific ones.

Step 1 - Identify risks by quality characteristic

Walk through ISO 25010 quality characteristics. For each, ask: "What could go wrong here in this product?"

Per ISO/IEC 25010:2023 quality model (cite by stable ID).

Aim for 15-30 product-level risks. Fewer than 10 = too sparse; more than 50 = mix with per-release risks.

Step 2 - Score per impact × likelihood

Use the same 1-5 scale as the spine's matrix. Score is impact × likelihood:

Step 3 - Document the register

# Product risk register - <product-name>

**Last reviewed:** YYYY-MM-DD  **Owner:** <name>  **Next review:** YYYY-MM-DD

## Active risks (n=22)

| ID | Category (ISO 25010) | Risk | Impact | Likelihood | Score | Mitigation(s) | Owner | Last review |
|---|---|---|---:|---:|---:|---|---|---|
| PR-001 | Functional suitability | Pricing engine off-by-cent in EU markets | 5 | 3 | 15 | Property-based testing on rounding; nightly compliance test suite | Alice | 2026-05-01 |
| PR-002 | Security | OAuth refresh-token leak via logs | 5 | 2 | 10 | Log redaction middleware; quarterly secret-scan; presidio-pii-detection in CI | Bob | 2026-04-15 |
| PR-003 | Reliability | Stripe webhook delivery failure not retried | 4 | 4 | 16 | DLQ + retry; chaos test in staging weekly | Carol | 2026-05-10 |
| PR-004 | Performance | Catalog search slows under >10k SKUs | 4 | 3 | 12 | Elasticsearch tuning; k6 load test gate | Dan | 2026-03-20 |
| ... | ... | ... | ... | ... | ... | ... | ... | ... |

## Retired risks (n=5)

| ID | Risk | Retired date | Why retired |
|---|---|---|---|
| PR-R-001 | Legacy SOAP API maintenance | 2026-02-01 | API decommissioned in v3.0 |
| ... | ... | ... | ... |

Step 4 - Link mitigations to test coverage

For each active risk, name at least one mitigation and link it to existing test coverage (or flag a gap):

| PR-001 | Pricing engine off-by-cent | 15 | Tests: tests/billing/test_promo_stacking.py + nightly compliance suite (PROJ-T123 + 124) | Alice |

Use the risk coverage mapping workflow in SKILL.md to generate the test-to-risk map.

Step 5 - Quarterly review

Per ISTQB CTAL-TM, product risks should be re-scored at least quarterly:

## Q2 2026 review log

- **PR-001**: Impact unchanged (5); likelihood **lowered 3 → 2**
  (property-based tests in place + 6 months no incidents). New
  score: 10 (was 15).
- **PR-003**: Impact unchanged (4); likelihood **raised 4 → 5**
  after Q1 webhook outage at provider. New score: 20 (was 16).
- **PR-007**: **Retired** - feature deprecated in v3.0.
- **New: PR-022**: Locale-related date formatting; impact 3,
  likelihood 3, score 9. Owner: Eve.

Net change: 1 new, 1 retired, 1 risen, 1 lowered. Total active: 22.

Step 6 - Seed the release matrix

When a release matrix starts:

1. Copy product risks scoring ≥10 (High + Critical tier) into the release matrix.
2. Identify release-specific risks beyond the persistent ones (per the release's feature set).
3. Assign the release-specific testing strategy per SKILL.md Step 4.

Worked example - e-commerce product

A 24-risk register might decompose like:

Distribution suggests Security + Functional suitability are the top areas to invest in.

Anti-patterns

Limitations

• Subjective scoring. Impact × likelihood depends on judgment; inter-rater agreement is moderate. Use the rubric anchors in SKILL.md Step 2.
• Quarterly cadence is a floor. Volatile products may need monthly; mature products may extend to annual.
• Doesn't replace per-feature analysis. Product risks are high-altitude; per-feature risk analysis happens at release.
• Mitigation linkage is one-directional. Test coverage may exceed risk mitigation (defensible: unconnected coverage isn't bad). Reverse - risks without coverage - needs explicit attention.

References

• ISTQB Advanced Test Manager (CTAL-TM) syllabus, ch. 5 on risk-based testing.
• ISO 31000:2018 "Risk management - Guidelines" - cite by stable ID; iso.org paywall.
• ISO/IEC 25010:2023 "Systems and software engineering - Systems and software Quality Requirements and Evaluation (SQuaRE) - Product quality model" - cite by stable ID.
• ISTQB Glossary - glossary.istqb.org (opens in new window) - "product risk", "project risk", "risk identification".
• Sibling reference (different scope): project-risk-register.md (opens in new window) - project-level: schedule, env, people.

Project risk register

Deep reference for the risk-matrix SKILL.md. The project-level register variant: risks to project execution (schedule slippage, environment instability, people / staffing, vendor / dependency, scope creep) rather than the product itself. The project manager reviews it weekly.

Per ISTQB CTAL-TM, the distinction is:

• Product risk - "the possibility that the system or software might fail to satisfy or fulfil some reasonable expectation of the customer, user, or stakeholder." → captured in product-risk-register.md (opens in new window) and per-release in the spine's matrix.
• Project risk - "any risk that affects project success." Schedule, resources, etc.

Cite ISTQB glossary (glossary.istqb.org (opens in new window)) "product risk" + "project risk". Project risks are typically excluded from the risk-to-test-coverage matrix (SKILL.md coverage mapping section): only product / release risks map to test coverage.

When to build one

• Sprint planning - capture sprint-specific project risks.
• Release planning - capture release execution risks (deadlines, dependencies, staffing).
• Quarterly OKR review - capture longer-horizon project risks.
• Project kickoff - establish the baseline.

Step 1 - Identify by category

Per ISO 31000:2018 (cite by stable ID) and PMI PMBOK 7th edition risk categorisation, project risks fall into broad buckets:

Aim for 8-20 active project risks at any time. Stale entries get retired.

Step 2 - Score per impact × likelihood

Impact 1-5:
  1 = Minor delay / inconvenience
  2 = Multi-day delay; small re-work
  3 = Sprint slippage; significant re-work
  4 = Release slippage; major re-plan
  5 = Quarter slippage; reputation / financial damage

Likelihood 1-5:
  1 = Very unlikely (<10% in horizon)
  2 = Unlikely (~25%)
  3 = Possible (~50%)
  4 = Likely (~75%)
  5 = Very likely (>90%)

Score = Impact × Likelihood (range 1-25)

Step 3 - Pick a mitigation strategy

Per ISO 31000:2018, four standard responses:

Document the chosen strategy per risk.

Step 4 - Document the register

# Project risk register - <project / quarter>

**Last reviewed:** YYYY-MM-DD  **Owner:** <PM>  **Next review:** YYYY-MM-DD

## Active risks (n=11)

| ID | Category | Risk | Impact | Likelihood | Score | Strategy | Mitigation | Owner | Status |
|---|---|---|---:|---:|---:|---|---|---|---|
| PJ-001 | Schedule | Q2 launch depends on Stripe migration; Stripe rate-limits doc work | 4 | 3 | 12 | Mitigate | Run migration in parallel with team-2 work | Alice | Active |
| PJ-002 | People | Senior payments engineer on parental leave Q2 | 4 | 5 | 20 | Mitigate | Knowledge-transfer sessions Q1; pair-coding | Bob | Active |
| PJ-003 | Vendor | Auth0 deprecating legacy SDK Q3 | 3 | 5 | 15 | Mitigate | Migrate to new SDK Q2 (1 sprint allocated) | Carol | Active |
| PJ-004 | Compliance | SOC 2 Type II audit Q3 - controls evidence collection backlog | 5 | 3 | 15 | Mitigate | Hire compliance contractor; start collection Q1 | Dan | Active |
| PJ-005 | Environment | Staging cluster autoscale unreliable | 3 | 4 | 12 | Mitigate | Migrate staging to k8s-autopilot Q2 | Eve | Active |
| PJ-006 | Scope | "AI feature" spec drifting in stakeholder reviews | 3 | 4 | 12 | Mitigate | Lock spec by end Q1; weekly stakeholder review | Fran | Active |
| PJ-007 | Schedule | Cyber-week launch window non-negotiable | 5 | 2 | 10 | Mitigate + Accept | Buffer + war-room ready | Alice | Active |
| ... | ... | ... | ... | ... | ... | ... | ... | ... | ... |

## Retired / triggered risks (n=4)

| ID | Risk | Outcome | Lessons |
|---|---|---|---|
| PJ-R-001 | DB migration vendor delay | Triggered Q1; mitigated by parallel path | Two-vendor strategy works |
| PJ-R-002 | Staffing - onboarding lag | Not triggered | Mitigation (early hire) effective |

Step 5 - Weekly review cadence

Project risks evolve fast. Weekly review at standup or PM syncpoint:

## Week of YYYY-MM-DD review

- **PJ-002** (key engineer leave): KT session 3/4 complete. Score
  unchanged.
- **PJ-003** (Auth0 SDK): migration started; likelihood drops 5 → 3.
  New score: 9 (was 15).
- **New PJ-012**: Cloud provider regional outage (us-east-1) on
  YYYY-MM-DD reminded us of single-region risk. Impact 4,
  likelihood 2, score 8. Strategy: monitor.
- **Triggered PJ-007** (Cyber-week launch): launched successfully;
  retire.

Step 6 - Escalation triggers

Risks scoring ≥15 trigger escalation:

## Escalations this week

- **PJ-002** (score 20): Engineering Director informed.
  Mitigation status: 3/4 KT sessions held; remaining pair-coding
  blocked on engineer availability.

Worked example - quarterly project risk distribution

For a 12-risk register in a Q2 release:

People + Schedule are dominant - staffing + dependency mitigations need most attention.

Anti-patterns

Limitations

• Project risks evolve fast. Weekly review is the floor; some risks need daily attention.
• Scoring is judgmental. Project context differs by team / industry / org maturity.
• Doesn't replace project plan. Risks supplement the plan; they don't replace it.
• "Transfer" via vendor contract is often theoretical - contract clauses without enforcement don't actually transfer risk.

References

• ISO 31000:2018 "Risk management - Guidelines" - cite by stable ID; iso.org paywall.
• PMI PMBOK Guide 7th edition - risk management chapter.
• ISTQB Advanced Test Manager (CTAL-TM) syllabus, ch. 5.
• ISTQB Glossary - glossary.istqb.org (opens in new window) - "project risk" entry.
• Sibling reference (different scope): product-risk-register.md (opens in new window).

Risk-register review checklist

Deep reference for the risk-matrix SKILL.md. A hygiene checklist for auditing a risk register (per-release matrix or product register) for assessment quality before release planning or a quarterly review. Blocks substandard risk assessments from driving release planning.

Inputs: the register under review, optionally the coverage matrix from the SKILL.md coverage mapping section, and the decisions folder for accepted risks. Output: per-risk findings + a single register-level verdict (pass / block / pass-with-caveats).

Check 1 - Field completeness

Every entry must have:

Check 2 - Independence of impact and likelihood

Impact + likelihood must be independently scored, not auto-equated:

def check_independence(risks):
    issues = []
    pairs = [(r["impact"], r["likelihood"]) for r in risks]
    diag_count = sum(1 for i, l in pairs if i == l)
    if len(pairs) > 5 and diag_count / len(pairs) > 0.7:
        issues.append(
            f"{diag_count}/{len(pairs)} risks have impact == likelihood. "
            "Likely auto-equated; force independent scoring."
        )
    return issues

If >70% of risks have impact == likelihood, the register is suspect (the same principle as scoring defect severity and priority independently).

Check 3 - Strategy discipline

For each "Accept" decision, verify a written acceptance-decision memo exists (who accepted, what evidence, expiry / revisit date). Any Accept without a linked decision document = BLOCK.

For each "Transfer" decision, verify the receiving party (insurance / vendor / SLA) is named. Without it, "Transfer" is hand-wave Accept.

Check 4 - Mitigation-to-coverage linkage

Build the coverage matrix per the SKILL.md coverage mapping section, then:

orphans_critical = [r for r in matrix if r["coverage_depth"] == 0 and r["score"] >= 15]
orphans_high = [r for r in matrix if r["coverage_depth"] == 0 and 10 <= r["score"] < 15]

• Critical-score orphans (≥15) = BLOCK
• High-score orphans (10-14) = warning
• Medium-score orphans (5-9) = info

Check 5 - Escalation evidence

Risks scoring ≥15 must have escalation evidence:

Without it, the register has under-escalated risks = caveat.

Check 6 - Review cadence

Per the register type:

If most entries' last_review is older than the threshold, the register is stale.

Check 7 - Verdict + report

# Risk-register audit - Q2 2026 release matrix - YYYY-MM-DD

**Risks audited:** 27 active + 4 retired
**Findings:** 6 critical, 9 warnings
**Verdict:** BLOCK - 3 critical findings require fix before release

## Critical (must fix before release planning)

| # | Risk ID | Finding |
|---|---|---|
| 1 | R-14 | Strategy "Accept" without linked decision document |
| 2 | R-22 | Score 20 (impact 5 × likelihood 4); coverage depth 0 (ORPHAN). No test, no monitor, no decision. |
| 3 | PR-009 | Score 16; last review 137 days ago (stale for product register; threshold 100 d) |

## Warnings

| Risk ID | Finding |
|---|---|
| R-08 | impact 3 = likelihood 3 = score 9; pattern repeats for 19/27 entries - likely auto-equated |
| PR-003 | Strategy "Transfer" but recipient not named |
| R-11 | Owner field empty |
| ... | ... |

## Coverage integration

- 4 of 27 risks are orphan (no coverage). Of these:
 - 2 critical (score ≥15) - BLOCK above
 - 2 low (score <10) - info

## Acceptance decisions integration

- 3 Accept decisions found. 2 have linked documents (R-08, R-12);
  1 missing (R-14 - BLOCK above)

## Review cadence

- Average `last_review`: 18 days
- Stale entries (>14 days for release matrix): 7 of 27
- Most-stale: PR-009 at 137 days

## Action items

1. **R-14**: Either author the acceptance decision memo or change
   strategy to Mitigate / Transfer.
2. **R-22**: Add at least one mitigation + one test before release.
3. **PR-009**: Re-review now; update `last_review` date.

After fixes, re-run this checklist.

Never-pass rules

A register review never returns "pass" when any of these hold:

• Any critical-score risk is orphan in coverage.
• Any "Accept" decision lacks a written decision document.
• More than 50% of entries are stale per the applicable cadence.
• Findings were suppressed without a per-risk waiver.

The review reports and recommends; it never auto-fixes register fields.

Anti-patterns

Limitations

• Subjective scoring. The checklist checks discipline, not correctness of impact / likelihood values.
• Cadence threshold is org-policy. Defaults from ISTQB CTAL-TM but teams may justify exceptions.
• Coverage check depends on the coverage-mapping output. If tags / refs aren't disciplined, coverage is under-reported; the review shows orphans that may actually be covered.
• Doesn't validate decisions. A documented Accept decision with bad rationale still passes the linkage check; pair with human review of decisions themselves.

References

• ISO 31000:2018 - cite by stable ID.
• ISTQB Advanced Test Manager (CTAL-TM) syllabus, ch. 5.
• Sibling references: product-risk-register.md (opens in new window), project-risk-register.md (opens in new window), calibration.md (opens in new window).

Risk storming - facilitation guide

Deep reference for the risk-matrix SKILL.md. How to plan and facilitate the risk-storming session that fills the matrix: meeting structure, participant roster, per-category brainstorm prompts, affinity grouping, impact × likelihood scoring, and mitigation assignment.

Risk-storming is a collaborative exercise where the team brainstorms what could go wrong with a feature or release. The output feeds directly into the risk matrix (per SKILL.md Step 2). The format originates from threat modeling exercises (notably Adam Shostack's Threat Modeling: Designing for Security) and Gojko Adzic's risk-driven testing community, adapted as a general QA technique.

When to run a session

• A new feature is about to enter development; the team needs to identify risks proactively.
• A release scope is set; pre-implementation risk review.
• Quarterly: the team revisits its risk register.
• Post-incident: a root cause investigation surfaces new risk categories worth brainstorming.

Session at a glance

1. Schedule 60-90 min and invite cross-functional participants - engineers, one QA / SDET, one PM, plus SRE / Security / Compliance when relevant; pre-distribute the spec, acceptance criteria, current matrix, and similar postmortems (Step 1).
2. Kick off by presenting the scope and reading the ACs aloud, then run a silent brainstorm where each participant lists 5-10 risks (Step 2).
3. Drive the brainstorm with the per-category prompts (Step 3).
4. Affinity-group the raw risks into clusters (Step 4).
5. Score each risk impact (1-5) x likelihood (1-5), capping debate at 5 min per risk (Step 5).
6. Assign a mitigation + owner + due date for every Critical (>=15) and High (9-14) risk (Step 6).
7. Write the results into the risk matrix, open trackers, and schedule a one-sprint check-in (Steps 7-8).

Step 1 - Pre-session setup

Schedule 60-90 min. Invite:

• 2-4 engineers (product code knowledge)
• 1 QA / SDET
• 1 PM (business context)
• Optional: SRE / Security / Compliance (if relevant)

Pre-distribute:

• The feature spec / story
• The acceptance criteria
• The current risk matrix (for the area, if exists)
• Recent incident postmortems for similar features

Step 2 - Session structure

00:00-00:05  Kickoff: facilitator presents the feature scope; reads ACs aloud
00:05-00:25  Silent brainstorm: each participant lists 5-10 risks (post-its or shared doc)
00:25-00:40  Affinity grouping: cluster risks by category
00:40-00:55  Score each risk: impact (1-5) × likelihood (1-5)
00:55-01:15  Mitigation discussion + owner assignment
01:15-01:30  Review + close: confirm action items

The silent-brainstorm phase is critical - without it, the loudest voice dominates and group-think hides real risks.

Step 3 - Prompts per category

The facilitator brings prompts to drive the brainstorm, one set per risk category (categories align with SKILL.md Step 3). The prompts are starters; the participants extend per the feature.

Business risks

• "What if the calculation is off by a cent?"
• "What if a customer applies the same code twice?"
• "What if the discount stacks differently than expected?"
• "What's the worst incorrect outcome the customer would see?"

Technical risks

• "What if the third-party API is down?"
• "What if the DB migration runs partially?"
• "What if two users hit this concurrently?"
• "What's the longest-running query under this feature?"

Regulatory / compliance risks

• "What if this stores PII?"
• "What if the user is in EU / California?"
• "Are there regional pricing requirements?"

UX risks

• "What if the user's network is 3G?"
• "What if the user is using a screen reader?"
• "What's the first-time user experience?"

Security risks

• "What if an attacker controls the input?"
• "What if a logged-in user accesses another user's data?"
• "Where's the auth boundary?"

Performance risks

• "What's the cold-start time?"
• "What's the per-request latency budget?"
• "What's the concurrent-user ceiling?"

Step 4 - Affinity grouping

After the silent brainstorm, cluster:

Cluster: "Payment failures"
- Stripe webhook delivery failure
- Stripe API rate limit
- Customer card declined mid-checkout
- Customer cancels mid-checkout (browser back button)

Cluster: "Promo math"
- Off-by-cent rounding
- Stack two promos
- Apply expired promo
- Apply promo to free-shipping order

Cluster: "EU compliance"
- VAT calculation
- GDPR data export
- Cookie consent

Cluster: "..."

Clusters reveal that some risks are different facets of one underlying issue (e.g., "payment failures" is one architectural concern; mitigations may apply across the cluster).

Step 5 - Score

Per cluster (or per-row if the cluster has heterogeneous risks):

The team agrees on each score via brief discussion (5 min cap per risk). When discussion exceeds the cap, the facilitator notes the disagreement and moves on; revisit after the session.

Step 6 - Mitigations + owners

For each Critical (>=15) and High (9-14) risk:

Lower-priority risks (Medium / Low) get logged but may not get immediate mitigations.

Step 7 - Output to the risk matrix

The session output flows directly into the matrix per SKILL.md Step 2. The matrix file becomes the canonical record; the session notes (silent brainstorm results, discussion points) get attached as appendix.

Step 8 - Post-session

Within 1 day:

• Author / update the risk matrix file.
• Create tracker tickets for each action item.
• Schedule a 15-min check-in 1 sprint later to verify mitigations shipped.

A risk-storming session without follow-up is wasted.

Worked example

A team is about to build checkout promo codes. In the kickoff session:

1. Silent brainstorm surfaces (among others) "off-by-cent rounding when a percentage promo is applied" and "two promos stacked on one order."
2. Affinity grouping puts both under a "Promo math" cluster.
3. Scoring rates off-by-cent rounding impact 5 (wrong charge, refunds, trust) x likelihood 3 = 15, which lands as Critical.
4. Mitigation assigns "property-based tests on the rounding function" to Alice, due 2026-05-15.
5. Output writes the cluster into the matrix as one row per risk, and a tracker ticket is opened for Alice's task.

The session ends with a scored, owned Critical risk and a matrix row - not a vague "we should test rounding" note.

Anti-patterns

Limitations

• Time investment. 60-90 min × ~5 people = 5-7.5 person-hours per session. Don't run weekly.
• Facilitator skill matters. A weak facilitator produces a weak session.
• Group-think risk. Even with silent brainstorm, group dynamics can suppress disagreement. Encourage explicit "I disagree" framing.
• Doesn't predict unknown unknowns. Risk storming addresses identified risk classes; new ones emerge from incidents.

References

• Adzic, G. on risk-driven testing - community references at gojko.net.
• Shostack, A., Threat Modeling: Designing for Security (2014) - foundational for the structured prompt approach.