openai-evals

Authors and runs OpenAI Evals - Python framework + registry for evaluating LLMs and LLM-backed systems with `oaieval {model} {eval-name}` CLI; supports template-based evals (Match / Includes / FuzzyMatch / ModelBasedClassify) defined in `evals/registry/evals/*.yaml` against JSONL data files in `evals/registry/data/`, plus custom Python eval classes implementing the Eval interface. Use when the user works with the openai/evals repo, needs the OpenAI-curated eval registry, or contributes new evals via PR to the registry.

Install with skills.sh (any agent)

npx skills add testland/qa --skill openai-evals

View source

openai-evals

Overview

Per oa-gh (opens in new window), a registry of YAML eval-specs lives under evals/registry/evals/, each pointing to a JSONL data file under evals/registry/data/ (Git-LFS managed). The oaieval CLI runs an eval against any completion-function-protocol model - either one of OpenAI's curated evals or a custom one registered by the team.

When to use

The team contributes evals upstream to OpenAI's registry.
The user needs the broad set of OpenAI-curated evals as a baseline.
A custom Python eval class is required (e.g., complex grading logic that doesn't fit YAML templates).
The team standardized on OpenAI Evals before alternatives like Promptfoo / DeepEval emerged.

For new projects without a registry-contribution motive, evaluate promptfoo-evaluation or deepeval-evaluation first - both have lower friction for non-OpenAI workflows.

Step 1 - Install

For running existing evals (per oa-gh (opens in new window)):

pip install evals

For contributing new evals (clone first, then editable install):

git clone https://github.com/openai/evals.git
cd evals
pip install -e .

The editable install is required to register new evals and access the full registry source.

Step 2 - Run an eval

Per github.com/openai/evals/blob/main/docs/run-evals.md (opens in new window):

oaieval gpt-3.5-turbo test-match

Pattern: oaieval <model> <eval-name>. Per oa-run (opens in new window):

"Any implementation of the CompletionFn protocol can be run against oaieval."

Eval names are "specified in the YAML files under evals/registry/evals" (oa-run (opens in new window)); implementations live in evals/elsuite.

Step 3 - Logging

Per oa-run (opens in new window):

"logging locally or to Snowflake will write to tmp/evallogs"

Override with --record_path /custom/path/. Logs are JSONL events "which can be inspected using a text editor or analyzed programmatically" (oa-run (opens in new window)).

Common flags (oa-run (opens in new window)):

--no-local-run - Snowflake DB logging
--record_path <dir> - output directory
oaieval --help - full CLI options

Step 4 - Eval templates (YAML-defined)

Eval templates avoid Python authoring for common evaluation patterns. The four built-in templates per oa-gh (opens in new window) (referenced in eval-templates.md):

Match - exact-match scoring: completion must equal an entry in ideal (single string or list)
Includes - substring scoring: completion must contain ideal text
FuzzyMatch - relaxed-match scoring: token-level overlap between completion and ideal
ModelBasedClassify - judge-model evaluates a completion (used for open-ended outputs where exact-match doesn't apply)

A registered eval YAML lives at evals/registry/evals/<name>.yaml and references a JSONL file at evals/registry/data/<name>/samples.jsonl. Each JSONL row contains the input prompt + the ideal field used by the template.

Step 5 - Custom Python evals

For grading logic beyond templates, subclass the Eval interface. The full pattern lives in docs/custom-eval.md and docs/build-eval.md in the oa-gh (opens in new window) repository - author per the doc when authoring, then register in the YAML registry.

Step 6 - CI integration

OpenAI Evals does not ship a first-party CI action. Pattern:

oaieval gpt-4 my-eval --record_path ./evallogs
# parse JSONL evallog for pass-rate; fail CI if below threshold
jq -s '[.[] | select(.spec) | .]' ./evallogs/<run>.jsonl  # extract spec + outcomes

For PR-comment integration, parse the events JSONL into a summary and post via gh CLI (no built-in action).

Anti-patterns

Anti-pattern	Why it fails	Fix
Pick `Match` template for open-ended generation	Exact-match always fails on creative outputs	Use `ModelBasedClassify` (Step 4)
Skip `--record_path` in CI	Logs land in `/tmp` and disappear between steps	Always pass `--record_path`
Custom Python eval without registry YAML	`oaieval` can't find it	Register the YAML alongside the Python class (Step 5)
Run on `gpt-3.5-turbo` only	Model-version drift; results not reproducible	Pin specific snapshot (e.g., `gpt-4-0613`)

Limitations

OpenAI-first design - non-OpenAI providers via the CompletionFn protocol work but require shim code; for multi-provider evals start with promptfoo-evaluation.
The registry is large but dated - many evals target older OpenAI models; check eval-spec freshness before contributing.
No first-party CI integration - assemble pass-rate gates manually.
JSONL log inspection requires familiarity with the events schema (per-event types: sampling, match, metrics).

References

oa-gh (opens in new window) - main repo + framework overview
oa-run (opens in new window) - oaieval CLI reference
docs/build-eval.md, docs/custom-eval.md, docs/eval-templates.md in oa-gh (opens in new window) - authoring details (load from repo when building a new eval)
promptfoo-evaluation, deepeval-evaluation - lower-friction alternatives for new projects

Related skills

deepeval-evaluation

Authors and runs DeepEval - pytest-native LLM eval framework with `LLMTestCase` (input + actual_output + expected_output + retrieval_context) and ~11 built-in metrics (G-Eval, Answer-Relevancy, Faithfulness, Contextual-Recall / Precision / Relevancy, Hallucination, Bias, Toxicity, Summarization, JSON-Correctness); runs via `deepeval test run {file.py}` with `assert_test()` per test or `evaluate()` for batch; integrates Confident-AI dashboard. Use when the user prefers pytest workflow, works with RAG and needs faithfulness/contextual metrics out-of-the-box, or wants a managed dashboard.

giskard-llm

Authors and runs Giskard LLM scans - adversarial test-case generation for LLM applications via `giskard.scan(model)` covering 7 vulnerability categories (hallucination, harmful_content, prompt_injection, sensitive_information_disclosure, stereotypes, robustness, basic_sycophancy); wraps any callable model behind `giskard.Model(model_predict, model_type="text_generation", ...)`; emits HTML report. Use when the user needs adversarial / red-team coverage on top of functional eval suites.

langfuse-tracing

Wires Langfuse tracing into LLM apps for production observability, monitoring, telemetry, and offline eval - instruments via `@observe` (Python) / `startActiveObservation` (TS) decorators that auto-capture inputs / outputs / timings / errors per generation; exposes `langfuse.update_current_span()` for metadata + cost / latency annotation; supports trace-bound scoring for eval datasets and prompt-as-code management. Use when the user needs to monitor, log, trace, or debug LLM API calls in production beyond pre-deploy eval, wants to add LLM observability tooling to an existing app, or wants to ship traces from production to an eval dataset for offline regression testing.

llm-eval-anti-patterns

Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy: too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check on paraphrase-tolerant output, no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage. Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape. Covers validating an LLM judge against human labels before its verdicts count as evidence. Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built.

llm-regression-suite-author

Builds a versioned golden-dataset LLM regression suite for tracking quality across model upgrades: structures a versioned JSONL/CSV golden dataset, configures deterministic eval runs (temperature 0, seed), wires assertion layers (exact, semantic similarity, LLM-as-judge, rubric), enforces a pass-rate threshold with diff reporting vs the baseline model, and gates CI on regression. Use when upgrading an LLM provider model and needing a repeatable before/after quality gate, or when a prompt regression suite must track output quality across model versions over time.

promptfoo-evaluation

Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`, performance `latency` / `cost`, custom `javascript` / `python`), runs `npx promptfoo eval`, views HTML report via `promptfoo view`, and integrates CI for regression gating. Use when the user runs Promptfoo, asks about prompt regression suites, or needs an eval-driven workflow for LLM-backed features.

ragas-evaluation

Authors and runs Ragas - RAG-pipeline evaluation framework with metrics organized into RAG (Faithfulness, Response Relevancy, Context Precision/Recall, Context Entities Recall, Noise Sensitivity), Natural Language Comparison (Factual Correctness, Semantic Similarity, BLEU/ROUGE/CHRF/Exact Match), Agents/Tool-Use (Topic Adherence, Tool Call Accuracy/F1, Agent Goal Accuracy), General Purpose (Aspect Critic, Rubrics-based Scoring), Nvidia (Answer Accuracy, Context Relevance, Response Groundedness), and Summarization. Use when the user evaluates a RAG pipeline (retriever + generator) and needs the deepest metric variety in the OSS LLM-eval space.