qa-ml-models
ML model testing: 7 skills (giskard-tests, deepchecks-tests, evidently-monitoring, fairlearn-fairness, model-performance-regression-gate, model-risk-evidence-matrix, notebook-ci-pipeline-author). Covers tabular + NLP + vision validation, drift monitoring with alert triage, group fairness with a risk-tiered promotion gate, per-prediction explainability, and the Jupyter notebook CI pipeline (papermill + nbval + testbook + nbstripout).
Install this plugin
/plugin install qa-ml-models@testland-qaPart of role bundle: qa-role-ai
qa-ml-models
ML model testing: vulnerability scanning, data validation, drift monitoring with alert triage, group fairness, and risk-tiered evidence gating. Seven skills covering Giskard (scan() + test catalog), Deepchecks (suites for data integrity / train-test / model evaluation), Evidently (drift monitoring + drift-alert triage playbook), Fairlearn (MetricFrame + Reductions mitigation), a retrain regression gate, the model-risk evidence matrix (risk tiering, fairness gating workflow, and Alibi Explain per-prediction explanation records), and the Jupyter notebook CI pipeline (papermill + nbval + testbook + nbstripout).
Components
| Type | Name | Description |
|---|---|---|
| Skill | giskard-tests | scan() for performance bias / data leakage / robustness / ethical issues; auto-generates test suites |
| Skill | deepchecks-tests | Data integrity, train-test validation, model evaluation suites - same checks across research / CI / production |
| Skill | evidently-monitoring | Reference-vs-current drift detection; PSI / KS / Wasserstein stat tests; production scheduling; drift-alert triage playbook (ranked root-cause hypotheses: schema change, pipeline bug, skew, seasonality, population shift) |
| Skill | fairlearn-fairness | MetricFrame group-disaggregated metrics; ExponentiatedGradient + ThresholdOptimizer mitigation |
| Skill | model-performance-regression-gate | CI gate that blocks a retrained model regressing on held-out metrics vs production. |
| Skill | model-risk-evidence-matrix | Assigns a model a risk tier and derives the fairness and explainability evidence that tier must produce; fairness gating workflow with promote / needs-work / block verdicts and refuse rules; Alibi Explain explanation-record authoring in references/. |
| Skill | notebook-ci-pipeline-author | Wires papermill (parameterized execution), nbval (output regression), testbook (function unit tests), and nbstripout into one GitHub Actions notebook CI pipeline, with a notebook PR review checklist (BLOCK / WARN / INFO); per-tool depth in references/. |
Install
/plugin marketplace add testland/qa
/plugin install qa-ml-models@testland-qaSkills
deepchecks-tests
Run Deepchecks suites (data integrity, train-test validation, model evaluation) on tabular / NLP / vision data + models. Pass `result.passed_conditions()` to CI to gate on regressions; the same checks run during research, CI, and production monitoring per the Deepchecks lifecycle posture. Use before training to catch train-test leakage and data-integrity defects in a tabular, NLP, or vision dataset, and to re-run the same suite on production samples to detect drift.
evidently-monitoring
Use Evidently OSS (100+ evaluation metrics, declarative testing API) to detect data drift, target drift, and model-performance regression, wired into CI as a gate (a Report run with include_tests) and into production monitoring as a continuous check; reports as HTML + JSON for both human review and pipeline assertions. Includes a drift-alert triage playbook: classify the fired alert's signal, rank root-cause hypotheses (upstream schema change, pipeline bug, training-serving skew, seasonality, genuine population shift), and pick rollback, retrain, quarantine, or alert re-tuning. Use when you need a drift or quality gate, a scheduled monitoring job, or a structured triage of a fired drift alert, for a tabular ML model. Built on the Evidently API specifically: for DeepChecks-based validation suites use deepchecks-tests instead.
fairlearn-fairness
Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`, then mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`). Wire group-disaggregated assertions into the model-evaluation gate. Use when a model's decisions affect people and a stakeholder, auditor, or regulation (ECOA, GDPR Art. 22, EU AI Act high-risk) requires evidence of per-group outcomes, or when someone reports the model treats a specific group worse.
giskard-tests
Test ML models with Giskard's scan() vulnerability detector + test catalog (performance, robustness, fairness, data leakage, ethical issues) for tabular and NLP models. Wrap a prediction function in giskard.Model + a DataFrame in giskard.Dataset; emit test suites that pass/fail in CI. Use when a trained tabular or NLP model is about to ship with no test suite of its own, or when a feature-engineering or hyperparameter change needs a pre-merge scan for newly introduced vulnerabilities.
model-performance-regression-gate
Computes held-out metrics (accuracy, F1, AUC, RMSE) for a retrained model and compares them against the current production model, failing promotion when any metric regresses beyond a configured tolerance. Adds per-segment checks via Deepchecks WeakSegmentsPerformance so a model that improves globally but regresses on a key slice is still blocked. Use when a retrained model is a candidate for promotion and the CI pipeline must enforce a per-metric pass/fail gate before the artifact is pushed to the model registry.
model-risk-evidence-matrix
Assigns an ML model to a low, medium, or high risk tier from what its predictions decide about people, then derives the fairness and explainability evidence that tier must produce: group metrics per declared sensitive feature, intersectional breakdowns with per-cell counts, vulnerability scan categories, a drift monitoring plan, and per-prediction explanation logs. Supplies conventional demographic parity difference bands, a per-vulnerability-category blocking table, evidence rules marking a bundle incomplete or self-contradicting, and a fairness gating workflow that walks a candidate's model card + evidence bundle to a promote / needs-work / block verdict with refuse rules; a reference covers producing the explanation records with Alibi Explain. Use when a model release candidate is up for promotion and someone must decide which fairness artifacts are mandatory, when a declared risk tier's evidence bundle must be checked against what the tier demands, or when the evidence review must gate the promotion.
notebook-ci-pipeline-author
The single home for Jupyter notebook testing: wires parameterized execution (papermill), output regression (nbval), function-level unit tests (testbook), output stripping (nbstripout), and artifact upload into one working GitHub Actions CI pipeline, with per-tool depth for papermill (parameters tag, CLI/API, sweeps) and nbval (strict/lax modes, per-cell markers, sanitize config) in references/. Includes a notebook PR review checklist covering untested notebooks, --nbval-lax misuse, hardcoded credentials, non-deterministic output cells, missing parameters tags, and committed outputs, with BLOCK / WARN / INFO severities and a BLOCK-or-PASS verdict. Use when notebooks must run as parameterized regression jobs in CI, when a repo ships .ipynb files whose outputs must stay stable, or when a PR that adds or modifies notebooks needs a structured quality review.