evidently-monitoring

Evidently is "an open-source Python library with over 40+ million downloads. It provides 100+ evaluation metrics, a declarative testing API, and a lightweight visual interface" per Evidently docs (opens in new window).

When to use

• Pre-deployment gate: assert no data/target drift between candidate-model evaluation set and the production reference.
• Production monitoring: scheduled job comparing yesterday's traffic vs the reference window.
• Triage tool: when a model misbehaves in prod, run an Evidently Report comparing the bad period to a known-good window.

How to use

1. pip install evidently into the CI or monitoring environment (Step 1).
2. Load a pinned reference dataset and the current dataset as pandas DataFrames (Step 2).
3. Run Report([DataDriftPreset()]) with run(reference_data=, current_data=) and save_html for review (Step 3).
4. For a CI gate, add include_tests=True, read .dict()["tests"], and raise SystemExit on any FAIL/ERROR (Step 4).
5. Add RegressionPreset / ClassificationPreset when prediction + target columns exist, to catch performance regression (Step 5).
6. Wrap the gated Report in a daily scheduled job comparing yesterday's traffic to the pinned reference, notifying on-call on failure (Step 6).
7. Tune the per-column drift method (psi, etc.) and severity tiers so critical drift blocks and minor drift only alerts (Step 4).

Step 1 - Install

pip install evidently

See the canonical install page at https://docs.evidentlyai.com/docs/setup/installation for the current install options (pip install evidently, plus the evidently[llm] extra).

Step 2 - Reference + current datasets

The standard pattern compares two datasets:

• Reference - known-good baseline (e.g., training data, last validated production window).
• Current - what you're checking (candidate model eval set, or current production traffic).

import pandas as pd

reference_df = pd.read_parquet("reference.parquet")
current_df = pd.read_parquet("current.parquet")

Step 3 - Run a drift Report

from evidently import Report
from evidently.presets import DataDriftPreset

# The current API takes the preset list positionally; run() with keyword
# args is unambiguous about which dataset is which (per [Evidently Report]).
report = Report([DataDriftPreset()])
my_eval = report.run(reference_data=reference_df, current_data=current_df)
my_eval.save_html("drift_report.html")

Result: HTML dashboard + structured JSON. Per Evidently docs (opens in new window), the preset bundles per-feature drift detection with sane defaults.

Step 4 - Gate CI on the drift tests

In the current Evidently API there is no separate TestSuite class. You enable per-column pass/fail tests by passing include_tests=True to the Report, then read each test's status from the result, per Evidently Report (opens in new window):

from evidently import Report
from evidently.presets import DataDriftPreset

# include_tests=True turns the preset's per-column drift metrics into
# pass/fail tests alongside the metrics.
report = Report([DataDriftPreset()], include_tests=True)
my_eval = report.run(reference_data=reference_df, current_data=current_df)

# .dict() exposes top-level "metrics" and "tests" only - there is NO
# top-level "status" key. Gate on any test that did not pass.
result = my_eval.dict()
failed = [t for t in result["tests"] if t.get("status") in ("FAIL", "ERROR")]
if failed:
    raise SystemExit(
        f"Evidently drift gate failed: {len(failed)} test(s); see drift_report.html"
    )

Evidently's drift detection supports several statistical methods (psi, wasserstein, ks, chisquare, jensenshannon); PSI is conventional for tabular production drift. Configure the method and threshold per column on the preset or the dataset's data definition, per Evidently drift preset (opens in new window).

Step 5 - Model-performance presets

from evidently.presets import RegressionPreset, ClassificationPreset

# Regression
report = Report([RegressionPreset()])
report.run(reference_data=ref, current_data=cur).save_html("regression.html")

# Classification
report = Report([ClassificationPreset()])
report.run(reference_data=ref, current_data=cur).save_html("classification.html")

Requires both prediction and target columns in both DataFrames.

Step 6 - Schedule in production

# Daily monitoring job
import datetime
from pathlib import Path

today = datetime.date.today().isoformat()
current_df = load_production_window(start=today, days=1)
reference_df = load_reference_window()

report = Report([DataDriftPreset()], include_tests=True)
result = report.run(reference_data=reference_df, current_data=current_df)
result.save_html(Path(f"monitoring/{today}.html"))

if any(t.get("status") in ("FAIL", "ERROR") for t in result.dict()["tests"]):
    notify_oncall(f"Data drift detected on {today}")

Pair with a scheduler (Airflow / Prefect / cron / Argo Workflows).

Worked example

A team ships a weekly retrain of a tabular fraud classifier and wants CI to block a release whose input distribution has moved too far from the validated baseline.

1. reference.parquet holds the last validated production week (~400k rows); current.parquet holds the candidate model's eval slice.
2. The CI step runs a gated drift Report:

from evidently import Report
from evidently.presets import DataDriftPreset

report = Report([DataDriftPreset()], include_tests=True)
result = report.run(reference_data=reference_df, current_data=current_df)
result.save_html("drift_report.html")

failed = [t for t in result.dict()["tests"] if t.get("status") in ("FAIL", "ERROR")]
if failed:
    raise SystemExit(f"Evidently drift gate failed: {len(failed)} test(s)")

1. Two columns (transaction_amount, merchant_country) drift past their PSI threshold, so their tests report FAIL.
2. raise SystemExit fails the CI job; drift_report.html is uploaded as a build artifact showing the two drifted distributions.
3. The engineer confirms merchant_country gained a new region, widens that column's threshold (or retrains on data that includes it), and the re-run passes.

Anti-patterns

Limitations

• 100+ metrics doesn't mean every domain. Healthcare/finance fairness metrics often need pairing with fairlearn-fairness skill.
• Memory: full preset on millions of rows can OOM. Sample to 100k - 1M before passing.

References

• Evidently docs (opens in new window) - library overview, presets, install snippet
• Evidently Report (opens in new window) - Report construction, include_tests, run(), and the .dict() result shape (top-level metrics + tests, per-test status)
• Evidently drift preset (opens in new window) - DataDriftPreset and per-column drift method configuration
• The Evidently docs (opens in new window) llms.txt index lists current canonical per-preset documentation: https://docs.evidentlyai.com/llms.txt