model-risk-evidence-matrix
Assigns an ML model to a low, medium, or high risk tier from what its predictions decide about people, then derives the fairness and explainability evidence that tier must produce: group metrics per declared sensitive feature, intersectional breakdowns with per-cell counts, vulnerability scan categories, a drift monitoring plan, and per-prediction explanation logs. Supplies conventional demographic parity difference bands, a per-vulnerability-category blocking table, evidence rules marking a bundle incomplete or self-contradicting, and a fairness gating workflow that walks a candidate's model card + evidence bundle to a promote / needs-work / block verdict with refuse rules; a reference covers producing the explanation records with Alibi Explain. Use when a model release candidate is up for promotion and someone must decide which fairness artifacts are mandatory, when a declared risk tier's evidence bundle must be checked against what the tier demands, or when the evidence review must gate the promotion.
Install with skills.sh (any agent)
npx skills add testland/qa --skill model-risk-evidence-matrixmodel-risk-evidence-matrix
This is a decision matrix for one question: given what a model decides, what fairness and explainability evidence must exist before anyone can reason about it? It converts a risk tier into a required-artifact list, and converts a missing or self-contradicting artifact into a named evidence rule.
What this owns, and what it does not
Owns: the risk-tiering scheme, the evidence-required-per-tier matrix, the conventional verdict bands on a group fairness metric, the per-vulnerability category blocking table, and the evidence rules that mark a bundle incomplete.
Does not own:
Step 1 - Assign the risk tier
Tier on what the prediction decides, not on model architecture, accuracy, or how the team feels about the use case.
| Tier | What puts a model here | Reversible by the affected person? | Example |
|---|---|---|---|
| Low | Internal recommendation consumed by staff; no decision about an identified individual | Yes | ranking an internal triage queue |
| Medium | External recommendation that shapes a user's experience; still no decision about the individual | Yes | storefront product recommender |
| High | Decides about an individual in credit, employment, healthcare, insurance, justice, or education | Usually not | consumer credit approval scoring |
Three external tests escalate a model to high regardless of where the team first placed it. Read each at its source before relying on it:
None of these three is a complete legal analysis. They are escalation triggers that force the high tier and send the question to counsel.
Step 2 - Read the required evidence for that tier
| Evidence artifact | Low | Medium | High |
|---|---|---|---|
| Performance metrics (aggregate) | required | required | required |
| Group fairness metrics per declared sensitive feature | - | required | required |
| Intersectional fairness across 2+ sensitive features | - | required | required |
| Model vulnerability scan | required | required | required |
| Drift monitoring plan with a declared reference dataset | - | required | required |
| Per-prediction (local) explanation logging | - | - | required |
| Mitigation provenance when any measured disparity is non-zero | - | required | required |
A - means the tier does not compel the artifact. It never means the artifact is unwelcome.
The declared sensitive-feature list is the spine of rows 2, 3, and 7. A model card that lists no sensitive features can only satisfy the low tier.
Step 3 - Name the fairness metric precisely
These three metrics are commonly used interchangeably in review conversations. They are not interchangeable. Each is defined below from Fairlearn's common fairness metrics guide (opens in new window).
| Metric | Fairlearn's definition | What it equalizes across groups |
|---|---|---|
| Demographic parity | "ensure a machine learning model's predictions are independent of membership in a sensitive group" | selection rate |
| Equalized odds | "ensure a machine learning model performs equally well for different groups" | true positive rate and false positive rate |
| Equal opportunity | "a relaxed version of equalized odds that only considers conditional expectations with respect to positive labels (Y=1)" | true positive rate only |
The conditions Fairlearn states for a classifier h with sensitive feature A and label Y, from the same guide:
demographic parity: E[h(X) | A=a] = E[h(X)] for all a
equalized odds: E[h(X) | A=a, Y=y] = E[h(X) | Y=y] for all a, y
equal opportunity: the equalized-odds condition restricted to Y=1The scalar reported against demographic parity is demographic_parity_difference, "defined as the difference between the largest and the smallest group-level selection rate", where a value of 0 "means that all groups have the same selection rate" (Fairlearn API reference (opens in new window)). This document abbreviates it DPD. Note the shape: DPD is a difference between rates. demographic_parity_ratio is a different quantity. Do not compare a difference against a threshold written for a ratio.
Which metric a bundle should carry depends on the harm being guarded against. Fairlearn's guidance: demographic parity where input biases are known; equalized odds where the historical data lacks measurement bias and both error types matter equally; equal opportunity where false positives cause less harm than missed true positives (Fairlearn, common fairness metrics (opens in new window)). Record the choice and its reason in the model card. An unexplained metric choice is a gap in the bundle, because the metric can be picked after the fact to flatter the result.
Step 4 - Accept that the metrics conflict
Do not ask a bundle to satisfy every fairness criterion at once. It is a proven result, not a tooling limitation, that common criteria conflict:
Evidence consequence: a bundle claiming to satisfy demographic parity and equalized odds and calibration across groups, on data with differing base rates, is making a claim the literature says is unavailable. Treat it as a reporting error to investigate, not as an unusually good result.
Step 5 - Apply the verdict bands, and label them as convention
| DPD on the declared sensitive feature | Band | What the bundle must then contain |
|---|---|---|
| DPD <= 0.05 | PASS | the measurement itself |
| 0.05 < DPD <= 0.10 | REVIEW | a written justification plus a monitoring plan |
| DPD > 0.10 | BLOCK | mitigation provenance, or a signed waiver |
These three numbers are a practitioner convention, not a standard and not a legal threshold. No standards body publishes 0.05 or 0.10 as a fairness limit. They are a starting policy for teams that have none, deliberately set tighter than any enforcement heuristic so that the review happens before a regulator's does.
The nearest well-known regulatory heuristic is the four-fifths rule, and even that is expressly not a legal test: the EEOC states the "'4/5ths' or '80%' rule of thumb is not intended as a legal definition, but is a practical means of keeping the attention of the enforcement agencies on serious discrepancies in rates of hiring, promotion and other selection decisions" (EEOC Q&A on the Uniform Guidelines (opens in new window)). It is also a ratio between selection rates, so it does not translate into a DPD difference threshold.
Who owns the numbers. Not the person running the metrics, and not this matrix. The band belongs to a named accountable owner recorded in the model card: legal counsel for a regulated decision, otherwise the risk or product owner who accepts the consequence. Fairlearn frames fairness assessment as requiring stakeholder identification and human judgment in context rather than a technical fix (Fairlearn, fairness in machine learning (opens in new window)), and the NIST AI Risk Management Framework is "intended for voluntary use" rather than as a prescriptive threshold source (NIST AI RMF (opens in new window)). A band with no named owner is an unowned band: record that as a gap.
A waiver, where used, carries four fields and no fewer: Reason:, Approved-by:, Re-review-date:, expires:. A waiver without an expiry is a silent permanent exemption.
Step 6 - Require intersectional evidence with counts
For medium and high tiers, single-attribute fairness is not sufficient evidence. Disparities that cancel out within sex and within race separately can still be severe at sex x race.
Fairlearn supports this by accepting multiple sensitive features, after which "the MetricFrame.by_group property then holds the intersections of these groups". The same guide warns that "some of the intersections may have very few members (or even be empty)", that an empty intersection shows as NaN, and that "random noise has a greater effect on smaller groups". It recommends including count() as a metric alongside the fairness metrics (Fairlearn, intersecting groups (opens in new window)).
Evidence rule: an intersectional table without a per-cell count column is not intersectional evidence. Its small cells cannot be distinguished from noise, and its NaN cells cannot be distinguished from unmeasured ones.
Step 7 - Triage vulnerability scan categories
A scan of the model surfaces categories of vulnerability rather than a single score. Giskard's scan reports performance bias, unrobustness, overconfidence, underconfidence, unethical behaviour, data leakage, stochasticity, and spurious correlation (Giskard tabular scan (opens in new window)).
This matrix assigns a blocking rule to six of them:
| Scan category | Blocks the bundle? | Why |
|---|---|---|
| Performance bias on a sensitive feature | YES | same disparity as Step 5, found on a different path |
| Data leakage | YES | training contamination invalidates every metric in the bundle |
| Unethical behaviour | YES | needs a human review record, not an automated pass |
| Unrobustness | Depends on input source: block when the input is user-controlled | an adversary can reach a user-controlled input |
| Underconfidence | NO | advisory; record it, do not gate on it |
| Stochasticity | NO when reproducible runs are configured; otherwise treat as unresolved | non-reproducible runs make every other finding unrepeatable |
Overconfidence and spurious correlation carry no blocking rule here. Record them and route them to whoever owns model quality.
Step 8 - Check the evidence rules
Each rule below marks a bundle incomplete (a required artifact is absent) or contradicted (the bundle asserts something its own artifacts do not support).
| Rule | Trigger | Status |
|---|---|---|
| R1 | Tier is medium or high and the declared sensitive-feature list is empty or ["none"] | incomplete: rows 2, 3 and 7 of the Step 2 matrix cannot be evaluated at all |
| R2 | Tier is high and no per-prediction explanation logs exist | incomplete: the artifact Step 2 marks required for high is absent |
| R3 | Any DPD exceeds 0.10 with no mitigation provenance and no waiver carrying all four fields | incomplete: the Step 5 BLOCK band names an artifact that is missing |
| R4 | The model card claims production drift monitoring but no schedule, reference dataset, or alert route exists | contradicted: a claim without its artifact |
| R5 | The vulnerability scan reports data leakage | contradicted: every performance and fairness number in the bundle is computed on compromised data |
| R6 | Tier is medium or high and no intersectional table exists, or it exists without per-cell counts | incomplete: see Step 6 |
R4 depends on a real reference dataset, not a nominal one. Evidently's drift preset requires both datasets to be supplied: "You must always pass both: the current one will be compared to the reference" (Evidently data drift preset (opens in new window)). A monitoring plan naming no reference dataset cannot run.
R2 depends on local explanations, meaning explanations of individual predictions. Explainability insights divide into local and global, where local insights concern individual predictions; Anchors finds "which set of features of a given instance are sufficient to ensure the prediction stays the same" and counterfactuals find "what minimal change to features is required to reclassify the current prediction" (Alibi Explain (opens in new window)). A global feature-importance chart does not satisfy R2, because the ECOA requirement in Step 1 is a reason for this applicant's decision. Producing the explanation records themselves - explainer selection, the fit/explain interface, and audit-log persistence - is covered in references/alibi-explainability.md.
Step 9 - Gate a promotion on the matrix (fairness gating)
The matrix doubles as a promotion gate. Given a release candidate's model card (risk tier, sensitive features, training data source, intended use) and its evidence bundle (Fairlearn MetricFrame.by_group numbers, vulnerability scan JSON, drift monitoring plan, explanation samples), walk the steps above in order and return one verdict: promote, needs-work, or block.
Refuse-to-promote rules. Never emit promote while any of these holds:
The gate verdict is about the evidence, not the model: promote means the bundle is complete and internally consistent for the tier. The release decision itself stays with the accountable owner named in Step 5.
Worked example
A full walk-through - a credit model mis-declared as medium, re-tiered to high, with rules R2, R4, and R6 firing and the emitted evidence-coverage report - is in references/worked-example.md.
Anti-patterns
| Anti-pattern | Why it fails | Instead |
|---|---|---|
| Tier on model architecture or accuracy | Tier is set by what the prediction decides about a person, not by how the model works | Step 1 |
| Treat aggregate accuracy as fairness evidence | Aggregates hide per-group disparity by construction | require the Step 2 group rows |
| Use "demographic parity" and "equalized odds" as synonyms | They equalize different quantities and can conflict | Step 3 definitions |
| Demand every fairness criterion at once | Proven unavailable outside constrained special cases | Step 4 |
| Present 0.05 / 0.10 as a legal or standards threshold | No standards body publishes them; the nearest heuristic is explicitly not a legal definition | label as convention, name an owner (Step 5) |
| Compare a difference metric against the 80% ratio rule | Difference and ratio are different quantities | pick one shape and state it (Step 3) |
| Single sensitive attribute only | Cancels out intersectional disparity | Step 6 |
| Intersectional table without counts | Small cells are indistinguishable from noise | Step 6 |
| Skip explanation logging because the model class is "interpretable" | Adverse-action rules require a reason for the individual decision, not a model property | Step 8, R2 |
| Waiver with no expiry | Becomes a permanent silent exemption | four-field waiver (Step 5) |
Limitations
Alibi Explain - producing the explanation evidence
View source (opens in new window)Alibi Explain - producing the explanation evidence
How to produce the per-prediction explanation artifacts the matrix requires (Step 2 row 6, rule R2) with Alibi Explain - Anchors, Integrated Gradients, Kernel/Tree SHAP, ALE, Counterfactual Instances.
Alibi Explain provides explanation algorithms answering: "How do predictions change with feature inputs? Which features matter for a prediction? What minimal changes would alter a prediction? How does each feature contribute to predictions?" per the Alibi Explain docs (opens in new window).
When to use
Step 1 - Install
pip install alibiPer the Alibi Explain docs (opens in new window).
Step 2 - Pick the right explainer category
Per the Alibi Explain docs (opens in new window):
| Category | Explainers | When |
|---|---|---|
| Global feature attribution | Accumulated Local Effects (ALE), Partial Dependence | "Across the whole input space, how does feature X drive output?" |
| Local necessary features | Anchors, Pertinent Positives | "What minimal feature subset locks in this prediction?" |
| Local feature attribution | Integrated Gradients, Kernel SHAP, Tree SHAP | "What did each feature contribute to this prediction?" |
| Counterfactual | Counterfactual Instances, CEM, CFProto, CounterfactualRL | "What minimal change flips this prediction?" |
Step 3 - The two-method interface
Every Alibi explainer follows the same pattern:
explainer.fit(X_train) # Some explainers - preparation phase
explanation = explainer.explain(instance)
print(explanation.data) # Per-explainer schemaPer the Alibi Explain docs (opens in new window) Explainer Interface section.
Step 4 - Anchors example (tabular)
from alibi.explainers import AnchorTabular
predict_fn = lambda x: classifier.predict(x)
explainer = AnchorTabular(
predict_fn,
feature_names=FEATURE_NAMES,
categorical_names=CATEGORICAL_INDEX,
)
explainer.fit(X_train)
explanation = explainer.explain(X_test[0])
print("Anchor: %s" % (" AND ".join(explanation.anchor)))
print("Precision: %.2f" % explanation.precision)
print("Coverage: %.2f" % explanation.coverage)Anchors return a minimal feature subset such that the prediction holds with precision confidence over coverage of the input space.
Step 5 - Counterfactual example
from alibi.explainers import CounterfactualProto
cf = CounterfactualProto(
predict_fn,
shape=X_train[0:1].shape,
use_kdtree=True,
theta=10.,
)
cf.fit(X_train)
explanation = cf.explain(X_test[0:1])
print("Counterfactual: %s" % explanation.cf["X"])
print("Original class: %d, CF class: %d" % (
explanation.orig_class, explanation.cf["class"]
))Counterfactual = "the closest input that flips the prediction" - auditor-friendly format.
Step 6 - Persist explanations as audit records
import json
from pathlib import Path
def explain_and_log(instance_id, instance, explainer, log_dir="explanations"):
explanation = explainer.explain(instance)
record = {
"instance_id": instance_id,
"timestamp": "...",
"explainer": type(explainer).__name__,
"data": explanation.data,
"meta": explanation.meta,
}
Path(log_dir, f"{instance_id}.json").write_text(json.dumps(record))For high-risk systems, store every explanation alongside the prediction (immutable audit log). Pair with an audit-trail test skill for storage assertions.
Step 7 - Don't confuse with alibi-detect
Alibi Explain is the explanation library. Drift detection uses the sister package alibi-detect (pip install alibi-detect) - that covers concept drift, adversarial detection, outlier detection. They share governance but are separate packages.
Anti-patterns
| Anti-pattern | Why it fails | Fix |
|---|---|---|
| Use Kernel SHAP on every prediction in real time | O(n) model calls per explanation; latency-killer | Tree SHAP for tree models; cache for repeated instances |
| Show feature attributions to non-technical stakeholders | "0.3 contribution from 'income'" is jargon | Use Counterfactuals (Step 5) - natural-language friendly |
Skip fit() step | Some explainers (Anchors) need training data summary | Always fit on representative data (Step 3) |
| Treat explanation as ground-truth causality | Attributions are model-relative, not causal | Document this in audit trail metadata |
Mix alibi and alibi-detect packages | Different scope; same install string causes confusion | Install both explicitly when needed (Step 7) |
Limitations
References
Worked example - model-risk-evidence-matrix
View source (opens in new window)Worked example - model-risk-evidence-matrix
A walk-through of a mis-tiered credit model checked against the evidence matrix. Step numbers and rule IDs (R2, R4, R6) refer back to SKILL.md.
Input. Model card declares: consumer credit approval scoring, tier stated as medium, sensitive features ["sex", "race"], "monitored in production". Bundle contains aggregate metrics, a group fairness table on sex and on race, and a vulnerability scan.
Step 1. The stated tier is wrong. Credit approval decides about an individual in credit, and Regulation B requires specific principal reasons for an adverse action (12 CFR 1002.9 (opens in new window)). Re-tier to high. Every subsequent step uses the high column.
Step 2 gap list. Intersectional table: absent. Per-prediction explanation logs: absent. Drift monitoring plan: claimed, artifact not present.
Step 3. The bundle reports DPD only. That is a defensible choice for credit selection, but the model card gives no reason for it, so the choice itself is recorded as a gap.
Step 5. DPD on sex is 0.087, landing in the REVIEW band, so a written justification plus a monitoring plan must be present. DPD on race is 0.04, PASS.
Steps 6 and 8. R6 fires (no intersectional table). R2 fires (no explanation logs). R4 fires (production monitoring claimed, no schedule or reference dataset).
Output.
## Evidence coverage - `credit-approval-v4`
Tier as declared: medium
Tier as assessed: HIGH (individual credit decision; adverse-action reasons required)
Sensitive features declared: sex, race
Bundle status: INCOMPLETE (3 rules fired)
| Required artifact (high tier) | Present | Note |
|---|---|---|
| Performance metrics | yes | accuracy 0.86, AUC 0.89 |
| Group fairness: sex | yes | DPD 0.087 -> REVIEW band, justification missing |
| Group fairness: race | yes | DPD 0.04 -> PASS |
| Intersectional (sex x race, with counts) | no | R6 |
| Vulnerability scan | yes | 0 blocking categories, 2 underconfidence findings (advisory) |
| Drift monitoring plan | claimed | R4: no schedule, no reference dataset |
| Per-prediction explanation logs | no | R2 |
| Mitigation provenance (DPD > 0) | no | required once a disparity is non-zero |
### Rules fired
- R2 - high tier, no local explanation logging.
- R4 - "monitored in production" asserted with no reference dataset or schedule.
- R6 - no intersectional table across the two declared sensitive features.
### Unowned decisions
- No named owner recorded for the 0.05 / 0.10 DPD bands.
- No recorded reason for choosing demographic parity over equalized odds.
### To close the bundle
1. Produce an intersectional table across sex x race including a per-cell count column.
2. Log local explanations for at least one positive and one negative predicted class.
3. Attach the drift schedule plus the named reference dataset, or drop the monitoring claim.
4. Attach mitigation provenance or a four-field waiver for the 0.087 DPD on sex.
5. Record the band owner and the metric-choice rationale in the model card.The output never says the model is fair or unfair. It says which required evidence exists and where the bundle contradicts itself.
Related skills
deepchecks-tests
Run Deepchecks suites (data integrity, train-test validation, model evaluation) on tabular / NLP / vision data + models. Pass `result.passed_conditions()` to CI to gate on regressions; the same checks run during research, CI, and production monitoring per the Deepchecks lifecycle posture. Use before training to catch train-test leakage and data-integrity defects in a tabular, NLP, or vision dataset, and to re-run the same suite on production samples to detect drift.
evidently-monitoring
Use Evidently OSS (100+ evaluation metrics, declarative testing API) to detect data drift, target drift, and model-performance regression, wired into CI as a gate (a Report run with include_tests) and into production monitoring as a continuous check; reports as HTML + JSON for both human review and pipeline assertions. Includes a drift-alert triage playbook: classify the fired alert's signal, rank root-cause hypotheses (upstream schema change, pipeline bug, training-serving skew, seasonality, genuine population shift), and pick rollback, retrain, quarantine, or alert re-tuning. Use when you need a drift or quality gate, a scheduled monitoring job, or a structured triage of a fired drift alert, for a tabular ML model. Built on the Evidently API specifically: for DeepChecks-based validation suites use deepchecks-tests instead.
fairlearn-fairness
Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`, then mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`). Wire group-disaggregated assertions into the model-evaluation gate. Use when a model's decisions affect people and a stakeholder, auditor, or regulation (ECOA, GDPR Art. 22, EU AI Act high-risk) requires evidence of per-group outcomes, or when someone reports the model treats a specific group worse.
giskard-tests
Test ML models with Giskard's scan() vulnerability detector + test catalog (performance, robustness, fairness, data leakage, ethical issues) for tabular and NLP models. Wrap a prediction function in giskard.Model + a DataFrame in giskard.Dataset; emit test suites that pass/fail in CI. Use when a trained tabular or NLP model is about to ship with no test suite of its own, or when a feature-engineering or hyperparameter change needs a pre-merge scan for newly introduced vulnerabilities.
model-performance-regression-gate
Computes held-out metrics (accuracy, F1, AUC, RMSE) for a retrained model and compares them against the current production model, failing promotion when any metric regresses beyond a configured tolerance. Adds per-segment checks via Deepchecks WeakSegmentsPerformance so a model that improves globally but regresses on a key slice is still blocked. Use when a retrained model is a candidate for promotion and the CI pipeline must enforce a per-metric pass/fail gate before the artifact is pushed to the model registry.
notebook-ci-pipeline-author
The single home for Jupyter notebook testing: wires parameterized execution (papermill), output regression (nbval), function-level unit tests (testbook), output stripping (nbstripout), and artifact upload into one working GitHub Actions CI pipeline, with per-tool depth for papermill (parameters tag, CLI/API, sweeps) and nbval (strict/lax modes, per-cell markers, sanitize config) in references/. Includes a notebook PR review checklist covering untested notebooks, --nbval-lax misuse, hardcoded credentials, non-deterministic output cells, missing parameters tags, and committed outputs, with BLOCK / WARN / INFO severities and a BLOCK-or-PASS verdict. Use when notebooks must run as parameterized regression jobs in CI, when a repo ships .ipynb files whose outputs must stay stable, or when a PR that adds or modifies notebooks needs a structured quality review.