deepchecks-tests

Run Deepchecks suites (data integrity, train-test validation, model evaluation) on tabular / NLP / vision data + models. Pass `result.passed_conditions()` to CI to gate on regressions; the same checks run during research, CI, and production monitoring per the Deepchecks lifecycle posture. Use before training to catch train-test leakage and data-integrity defects in a tabular, NLP, or vision dataset, and to re-run the same suite on production samples to detect drift.

Install with skills.sh (any agent)

npx skills add testland/qa --skill deepchecks-tests

View source

deepchecks-tests

Deepchecks is "a holistic open-source solution for all of your AI & ML validation needs" per the Deepchecks welcome (opens in new window). Validates data integrity, train-test splits, model evaluation, end-to-end model development from research through production.

When to use

Pre-training: validate data integrity + train-test leakage.
Pre-deployment: model evaluation suite as merge gate.
Production monitoring: same suite re-run on production samples to detect drift.

Step 1 - Install

pip install deepchecks

Per the Deepchecks welcome (opens in new window) page.

Step 2 - Wrap data

For tabular models:

from deepchecks.tabular import Dataset

train_ds = Dataset(
    train_df,
    label="target",
    cat_features=CATEGORICAL_COLUMNS,
)
test_ds = Dataset(
    test_df,
    label="target",
    cat_features=CATEGORICAL_COLUMNS,
)

cat_features matters for distribution checks. The Vision and NLP APIs differ - see the Deepchecks welcome (opens in new window) section linking quickstarts for each data type.

Step 3 - Run the data integrity suite

from deepchecks.tabular.suites import data_integrity

integrity = data_integrity()
result = integrity.run(train_ds)
result.save_as_html("data_integrity.html")

Catches: duplicate rows, missing values, mixed types, conflicting labels, single-value features, string mismatches.

Step 4 - Run the train-test validation suite

from deepchecks.tabular.suites import train_test_validation

validation = train_test_validation()
result = validation.run(train_ds, test_ds)
result.save_as_html("train_test_validation.html")

Catches: target drift, feature drift, train-test data leakage, label imbalance, dataset size mismatch.

Step 5 - Run the model evaluation suite

from deepchecks.tabular.suites import model_evaluation

evaluation = model_evaluation()
result = evaluation.run(train_ds, test_ds, model)
result.save_as_html("model_evaluation.html")

if not result.passed_conditions():
    raise SystemExit("Deepchecks model evaluation failed")

Catches: performance regression vs baseline, weak segments, calibration issues, prediction drift between train and test.

Step 6 - Per-check thresholds

from deepchecks.tabular.checks import FeatureDrift

check = FeatureDrift().add_condition_drift_score_less_than(
    max_allowed_categorical_score=0.2,
    max_allowed_numeric_score=0.1,
)
result = check.run(train_ds, test_ds)

if not result.passed_conditions():
    print(result.value)
    raise SystemExit("FeatureDrift failed threshold")

Each check has add_condition_* methods; chain them for per-check gating.

Step 7 - CI integration

- name: Deepchecks suite
  run: |
    python ml/deepchecks_suite.py
- name: Upload Deepchecks reports
  uses: actions/upload-artifact@v4
  with:
    name: deepchecks-reports
    path: "*.html"

Anti-patterns

Anti-pattern	Why it fails	Fix
Skip `cat_features`	All categorical checks misfire	Always specify (Step 2)
Skip `data_integrity` suite	Train on leaky / dup-heavy data	Run before `train_test_validation` (Step 3)
Block CI on every check	Hundreds of warning conditions; team disables	Define per-check thresholds (Step 6); gate Critical only
Re-run on the SAME test split each PR	Fixed split → fixed results; no drift detection	Use rolling/cross-validation splits
Reuse training data as "current" for production monitoring	Always passes drift; blind to real drift	Use real production samples

Limitations

Deepchecks' "default conditions" are tuned for textbook ML; they may be too strict for real-world skewed data. Tune via add_condition_* methods (Step 6).
Vision suite memory footprint: large image datasets need batched runs. See the Deepchecks welcome (opens in new window) vision quickstart link.

References

Deepchecks welcome (opens in new window) - overview, install, suite/check architecture, per-data-type quickstart links

Related skills

alibi-explainability

Generates model explanations with Alibi Explain - Anchors, Integrated Gradients, Kernel/Tree SHAP, ALE, Counterfactual Instances. Wires explainer.fit + explainer.explain into model-evaluation pipelines so that every flagged prediction ships with a "why" record auditors can reason about. Use when a model decision must be explainable to an auditor, regulator, or affected user, or when a support team cannot answer why a specific prediction was made.

evidently-monitoring

Use Evidently OSS (100+ evaluation metrics, declarative testing API) to detect data drift, target drift, and model-performance regression, wired into CI as a gate (a Report run with include_tests) and into production monitoring as a continuous check; reports as HTML + JSON for both human review and pipeline assertions. Use when you need a drift or quality gate, or a scheduled monitoring job, for a tabular ML model. Built on the Evidently API specifically: for DeepChecks-based validation suites use deepchecks-tests instead.

fairlearn-fairness

Compute group fairness metrics (selection rate, demographic parity, equalized odds) per sensitive feature with `MetricFrame`, then mitigate disparities using Reductions algorithms (`ExponentiatedGradient` with constraint = `DemographicParity`/`EqualizedOdds`). Wire group-disaggregated assertions into the model-evaluation gate. Use when a model's decisions affect people and a stakeholder, auditor, or regulation (ECOA, GDPR Art. 22, EU AI Act high-risk) requires evidence of per-group outcomes, or when someone reports the model treats a specific group worse.

giskard-tests

Test ML models with Giskard's scan() vulnerability detector + test catalog (performance, robustness, fairness, data leakage, ethical issues) for tabular and NLP models. Wrap a prediction function in giskard.Model + a DataFrame in giskard.Dataset; emit test suites that pass/fail in CI. Use when a trained tabular or NLP model is about to ship with no test suite of its own, or when a feature-engineering or hyperparameter change needs a pre-merge scan for newly introduced vulnerabilities.

model-performance-regression-gate

Computes held-out metrics (accuracy, F1, AUC, RMSE) for a retrained model and compares them against the current production model, failing promotion when any metric regresses beyond a configured tolerance. Adds per-segment checks via Deepchecks WeakSegmentsPerformance so a model that improves globally but regresses on a key slice is still blocked. Use when a retrained model is a candidate for promotion and the CI pipeline must enforce a per-metric pass/fail gate before the artifact is pushed to the model registry.

model-risk-evidence-matrix

Assigns a machine learning model to a low, medium, or high risk tier from what its predictions decide about people, then derives the fairness and explainability evidence that tier must produce: group metrics per declared sensitive feature, intersectional breakdowns with per-cell counts, vulnerability scan categories, a drift monitoring plan, and per-prediction explanation logs. Supplies conventional demographic parity difference bands, a per-vulnerability-category blocking table, and evidence rules that mark a bundle incomplete or self-contradicting. Use when a model release candidate is up for promotion and someone must decide which fairness artifacts are mandatory rather than nice to have, or when a model card declares a risk tier and the attached evidence bundle has to be checked against what that tier demands.