deepeval-evaluation

Overview

Per de-start (opens in new window), DeepEval is "an open-source LLM eval package" enabling "evaluation of LLM applications locally through test cases and metrics." The model: each test constructs an LLMTestCase, applies one or more Metric instances, and either asserts (assert_test) or batch-evaluates (evaluate). Pytest discovery + reporting works unchanged.

When to use

• The repo already uses pytest; LLM tests should live alongside unit tests.
• The user works with RAG and needs Faithfulness / Contextual-* metrics without writing them from scratch.
• The team wants a managed dashboard (Confident-AI) for regression tracking + prompt-vs-prompt comparison.
• Programmatic test-case authoring (data-driven from a CSV/JSONL) is needed and pytest fixtures fit better than YAML config.

Step 1 - Install

Per de-gh (opens in new window) and de-start (opens in new window):

pip install -U deepeval

Optional Confident-AI login (for dashboard):

deepeval login

Per de-start (opens in new window): after login "Confident AI will generate testing reports and automate regression testing whenever you run a test run."

Step 2 - First test

Per de-gh (opens in new window) (verbatim quickstart):

import pytest
from deepeval import assert_test
from deepeval.metrics import GEval
from deepeval.test_case import LLMTestCase, SingleTurnParams

def test_case():
    correctness_metric = GEval(
        name="Correctness",
        criteria="Determine if the 'actual output' is correct based on the 'expected output'.",
        evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT, SingleTurnParams.EXPECTED_OUTPUT],
        threshold=0.5
    )
    test_case = LLMTestCase(
        input="What if these shoes don't fit?",
        actual_output="You have 30 days to get a full refund at no extra cost.",
        expected_output="We offer a 30-day full refund at no extra costs.",
        retrieval_context=["All customers are eligible for a 30 day full refund at no extra costs."]
    )
    assert_test(test_case, [correctness_metric])

Run it:

deepeval test run test_chatbot.py

(Per de-gh (opens in new window).)

Step 3 - LLMTestCase fields

Per de-start (opens in new window), LLMTestCase fields:

Step 4 - Metric catalog

Per de-gh (opens in new window) the available metrics include:

Each metric takes a threshold parameter; the test passes if score ≥ threshold.

Step 5 - Custom GEval pattern

GEval is the universal escape hatch when no built-in metric fits:

from deepeval.metrics import GEval
from deepeval.test_case import SingleTurnParams

professionalism = GEval(
    name="Professionalism",
    criteria="Determine if the response uses professional language without slang or contractions.",
    evaluation_params=[SingleTurnParams.ACTUAL_OUTPUT],
    threshold=0.7,
)

The criteria string is the rubric; the judge model evaluates and returns a 0 - 1 score with reasoning attached.

Step 6 - Batch evaluation (no pytest)

For dataset-driven runs without pytest:

from deepeval import evaluate

evaluate(test_cases=[case1, case2, case3], metrics=[g_eval, faithfulness])

Returns scores per metric per case; useful for regression sweeps across a CSV/JSONL of historical inputs.

Step 7 - CI integration

deepeval test run tests/llm/ --run-async --workers 4
# pytest exit code propagates: nonzero if any assert_test fails

Combine with Confident-AI for a dashboard view of run history; Confident-AI is the company behind DeepEval (per de-gh (opens in new window)).

Anti-patterns

Limitations

• LLM-as-judge metrics depend on judge-model quality + cost; pin judge model version in CI.
• Confident-AI is the managed dashboard; without it, regression tracking is manual (parse pytest output).
• Test cases live in Python files - not as discoverable as YAML configs for non-Python teammates (vs Promptfoo).
• Faithfulness / Contextual-* metrics need RAG-shaped data; for pure prompt evals, promptfoo-evaluation is lower-friction.

References

• de-gh (opens in new window) - repository, install, quickstart, metric list
• de-start (opens in new window) - getting-started overview
• promptfoo-evaluation, ragas-evaluation - sister tools (Promptfoo for YAML-config; Ragas for deeper RAG metric variety)