deepchecks-tests

Deepchecks is "a holistic open-source solution for all of your AI & ML validation needs" per the Deepchecks welcome (opens in new window). Validates data integrity, train-test splits, model evaluation, end-to-end model development from research through production.

When to use

• Pre-training: validate data integrity + train-test leakage.
• Pre-deployment: model evaluation suite as merge gate.
• Production monitoring: same suite re-run on production samples to detect drift.

Step 1 - Install

pip install deepchecks

Per the Deepchecks welcome (opens in new window) page.

Step 2 - Wrap data

For tabular models:

from deepchecks.tabular import Dataset

train_ds = Dataset(
    train_df,
    label="target",
    cat_features=CATEGORICAL_COLUMNS,
)
test_ds = Dataset(
    test_df,
    label="target",
    cat_features=CATEGORICAL_COLUMNS,
)

cat_features matters for distribution checks. The Vision and NLP APIs differ - see the Deepchecks welcome (opens in new window) section linking quickstarts for each data type.

Step 3 - Run the data integrity suite

from deepchecks.tabular.suites import data_integrity

integrity = data_integrity()
result = integrity.run(train_ds)
result.save_as_html("data_integrity.html")

Catches: duplicate rows, missing values, mixed types, conflicting labels, single-value features, string mismatches.

Step 4 - Run the train-test validation suite

from deepchecks.tabular.suites import train_test_validation

validation = train_test_validation()
result = validation.run(train_ds, test_ds)
result.save_as_html("train_test_validation.html")

Catches: target drift, feature drift, train-test data leakage, label imbalance, dataset size mismatch.

Step 5 - Run the model evaluation suite

from deepchecks.tabular.suites import model_evaluation

evaluation = model_evaluation()
result = evaluation.run(train_ds, test_ds, model)
result.save_as_html("model_evaluation.html")

if not result.passed_conditions():
    raise SystemExit("Deepchecks model evaluation failed")

Catches: performance regression vs baseline, weak segments, calibration issues, prediction drift between train and test.

Step 6 - Per-check thresholds

from deepchecks.tabular.checks import FeatureDrift

check = FeatureDrift().add_condition_drift_score_less_than(
    max_allowed_categorical_score=0.2,
    max_allowed_numeric_score=0.1,
)
result = check.run(train_ds, test_ds)

if not result.passed_conditions():
    print(result.value)
    raise SystemExit("FeatureDrift failed threshold")

Each check has add_condition_* methods; chain them for per-check gating.

Step 7 - CI integration

- name: Deepchecks suite
  run: |
    python ml/deepchecks_suite.py
- name: Upload Deepchecks reports
  uses: actions/upload-artifact@v4
  with:
    name: deepchecks-reports
    path: "*.html"

Anti-patterns

Limitations

• Deepchecks' "default conditions" are tuned for textbook ML; they may be too strict for real-world skewed data. Tune via add_condition_* methods (Step 6).
• Vision suite memory footprint: large image datasets need batched runs. See the Deepchecks welcome (opens in new window) vision quickstart link.

References

• Deepchecks welcome (opens in new window) - overview, install, suite/check architecture, per-data-type quickstart links