promptfoo-evaluation

Authors and runs Promptfoo evals for LLM prompts and RAG pipelines - wires `promptfooconfig.yaml` providers + prompts + tests + assertions (deterministic `equals` / `contains` / `is-json` / `regex`, semantic `similar`, model-graded `llm-rubric` / `factuality` / `g-eval`, performance `latency` / `cost`, custom `javascript` / `python`), runs `npx promptfoo eval`, views HTML report via `promptfoo view`, and integrates CI for regression gating. Use when the user runs Promptfoo, asks about prompt regression suites, or needs an eval-driven workflow for LLM-backed features.

Install with skills.sh (any agent)

npx skills add testland/qa --skill promptfoo-evaluation

View source

promptfoo-evaluation

A promptfooconfig.yaml declares providers (LLMs under test), prompts (templates with {{var}} placeholders), tests (input variables + assertions); promptfoo eval runs the cross-product (per pf-config (opens in new window)).

When to use

The repo has a promptfooconfig.yaml or the user wants to author one.
The user needs prompt regression suites - same inputs, multiple providers, fail-on-diff.
A CI workflow needs an eval gate on prompt or model changes.
The team prefers vendor-neutral eval (vs OpenAI Evals' OpenAI-first posture).

Step 1 - Install

Per github.com/promptfoo/promptfoo (opens in new window):

npm install -g promptfoo
# or
brew install promptfoo
# or one-off
npx promptfoo@latest

Per pf-gh (opens in new window): "Node.js 20.20+ or 22.22+ is required for npm and npx usage."

Step 2 - First eval

Initialize from a built-in example, then run:

promptfoo init --example getting-started
cd getting-started
promptfoo eval
promptfoo view  # opens HTML report

(Commands per pf-gh (opens in new window).)

A minimal promptfooconfig.yaml per pf-config (opens in new window):

prompts:
  - file://prompt1.txt

providers:
  - openai:gpt-5-mini
  - anthropic:claude-haiku-4-5

tests:
  - vars:
      language: French
      input: Hello world
    assert:
      - type: contains-json

Provider syntax <vendor>:<model> works for OpenAI, Anthropic, Vertex (vertex:gemini-2.0-flash-exp), Ollama (ollama:llama2), and 30+ others (per pf-config (opens in new window)).

Step 3 - Variable interpolation

Vars interpolate into prompts using {{variable}}. Arrays create combinations (per pf-config (opens in new window)):

tests:
  - vars:
      language: [French, German, Spanish]
      input: [Hello world, Good morning]

This generates 6 test rows (3 languages × 2 inputs). Vars can also load from files:

tests:
  - vars:
      var3: file://path/to/var3.txt
      context: file://fetch_from_vector_database.py

Step 4 - Assertion catalog

Per promptfoo.dev/docs/configuration/expected-outputs/ (opens in new window):

Deterministic (string + structure):

Type	Example
`equals`	`value: 'expected string'`
`contains` / `icontains`	`value: 'substring'` (case-sensitive / insensitive)
`starts-with`	`value: 'prefix'`
`contains-any` / `contains-all`	`value: ['a', 'b']`
`regex`	`value: '^pattern$'`
`is-json` / `contains-json`	(validates / locates JSON)
`is-sql` / `contains-sql`	(SQL syntax)
`is-xml` / `is-html` / `contains-xml` / `contains-html`	(markup)
`is-valid-openai-tools-call` / `is-valid-openai-function-call`	(tool calls)

Custom logic:

- type: javascript
  value: 'output.length > 10'
- type: python
  value: 'file://script.py'

Text-quality metrics (default thresholds per pf-asserts (opens in new window)):

Type	Default threshold
`rouge-n`	0.75
`bleu`	0.5
`gleu`	0.5
`meteor`	0.5
`levenshtein`	5 (edit distance max)

Performance + cost:

- type: latency
  threshold: 200    # milliseconds
- type: cost
  threshold: 0.001  # dollars per response

Model-graded (LLM-as-judge):

Type	Use
`llm-rubric`	Free-form rubric: `value: 'Is helpful and accurate'`
`model-graded-closedqa`	Closed-QA evaluation method
`factuality`	Compares against reference facts
`g-eval`	Chain-of-thought scoring
`answer-relevance`	Checks output relates to query
`context-faithfulness` / `context-recall` / `context-relevance`	RAG-specific

Semantic similarity:

- type: similar
  value: 'reference text'
  threshold: 0.8   # cosine similarity via embeddings

Negation: all deterministic assertions support a not- prefix (not-equals, not-contains, not-regex, etc.) per pf-asserts (opens in new window).

Step 5 - defaultTest pattern

Shared assertions/vars across all tests:

defaultTest:
  vars:
    shared_var: 'shared content'
  assert:
    - type: llm-rubric
      value: does not describe self as AI
  options:
    provider: openai:gpt-5-mini-0613

tests:
  - vars:
      unique_var: value1

(Per pf-config (opens in new window).)

Step 6 - Output transforms

Modify LLM output before assertions execute (per pf-config (opens in new window)):

tests:
  - vars:
      body: Hello world
    options:
      transform: output.toUpperCase()

Or load from a file:

options:
  transform: file://transform.js:customTransform

Step 7 - assert-set grouping

Group assertions and require a threshold percentage to pass (per pf-asserts (opens in new window)):

assert:
  - type: assert-set
    threshold: 0.5   # 50% of grouped asserts must pass
    assert:
      - type: cost
        threshold: 0.001
      - type: latency
        threshold: 200

Step 8 - CI integration

Provider API keys via env vars (per pf-gh (opens in new window): export OPENAI_API_KEY=sk-abc123).

GitHub Actions pattern (per promptfoo.dev/docs/integrations/github-action (opens in new window)):

- uses: promptfoo/promptfoo-action@v2
  with:
    openai-api-key: ${{ secrets.OPENAI_API_KEY }}
    config: 'promptfooconfig.yaml'
    cache-path: '~/.cache/promptfoo'
    use-config-prompts: false
    no-share: true
    promptfoo-version: 'latest'

The action posts a PR comment with regression diff vs the base branch. Caching reuses LLM responses for unchanged tests (per promptfoo.dev/docs/configuration/caching/ (opens in new window)).

Anti-patterns

Anti-pattern	Why it fails	Fix
Only deterministic assertions on creative outputs	LLM responses vary; rigid asserts produce flake	Use `llm-rubric` or `similar` (Step 4)
Single provider in config	Misses cross-provider regression	At least 2 providers per eval (Step 2)
No cost / latency caps	Eval cost balloons per PR	`cost` + `latency` asserts (Step 4)
`assert-set` with `threshold: 0`	Effectively disables grouping	Pick a real threshold (Step 7)
Fresh provider model on every run without pinning	Output drifts when vendor updates model	Pin specific snapshot (e.g., `openai:gpt-5-mini-0613`)
Skip caching in CI	Re-runs every test on every PR	`cache-path` in GHA action (Step 8)

Limitations

Model-graded assertions invoke a judge LLM per test row → cost per test ≈ 2× a deterministic-only eval.
Even seeded LLMs drift between provider model updates; pin model versions in CI to bound the regression surface.
Promptfoo ships HTML + JSON + JUnit reporters; no native Markdown PR-comment formatter (use the GHA action for that).

References

pf-intro (opens in new window) - overview
pf-config (opens in new window) - full configuration guide
pf-asserts (opens in new window) - assertion catalog
pf-gha (opens in new window) - GitHub Action
pf-cache (opens in new window) - caching mechanics
pf-gh (opens in new window) - install commands
openai-evals, deepeval-evaluation, ragas-evaluation - sister tools (different framework styles)

Related skills

deepeval-evaluation

Authors and runs DeepEval - pytest-native LLM eval framework with `LLMTestCase` (input + actual_output + expected_output + retrieval_context) and ~11 built-in metrics (G-Eval, Answer-Relevancy, Faithfulness, Contextual-Recall / Precision / Relevancy, Hallucination, Bias, Toxicity, Summarization, JSON-Correctness); runs via `deepeval test run {file.py}` with `assert_test()` per test or `evaluate()` for batch; integrates Confident-AI dashboard. Use when the user prefers pytest workflow, works with RAG and needs faithfulness/contextual metrics out-of-the-box, or wants a managed dashboard.

giskard-llm

Authors and runs Giskard LLM scans - adversarial test-case generation for LLM applications via `giskard.scan(model)` covering 7 vulnerability categories (hallucination, harmful_content, prompt_injection, sensitive_information_disclosure, stereotypes, robustness, basic_sycophancy); wraps any callable model behind `giskard.Model(model_predict, model_type="text_generation", ...)`; emits HTML report. Use when the user needs adversarial / red-team coverage on top of functional eval suites.

langfuse-tracing

Wires Langfuse tracing into LLM apps for production observability, monitoring, telemetry, and offline eval - instruments via `@observe` (Python) / `startActiveObservation` (TS) decorators that auto-capture inputs / outputs / timings / errors per generation; exposes `langfuse.update_current_span()` for metadata + cost / latency annotation; supports trace-bound scoring for eval datasets and prompt-as-code management. Use when the user needs to monitor, log, trace, or debug LLM API calls in production beyond pre-deploy eval, wants to add LLM observability tooling to an existing app, or wants to ship traces from production to an eval dataset for offline regression testing.

llm-eval-anti-patterns

Audits an existing LLM evaluation suite for eight methodology errors that make its numbers untrustworthy: too few cases per capability, single-provider lock-in, exact-match assertions on open-ended output, no semantic-similarity check on paraphrase-tolerant output, no baseline comparison in CI, no cost or latency ceiling, unpinned model identifiers, and no adversarial coverage. Supplies harness-neutral detection cues, the reason each error invalidates the result, a concrete fix, a Critical/Warning/Info severity scheme, and a findings-table output shape. Covers validating an LLM judge against human labels before its verdicts count as evidence. Use when an eval suite already exists and its pass rate is about to gate a release, a model swap, or a prompt change, and nobody has audited how the suite itself was built.

llm-regression-suite-author

Builds a versioned golden-dataset LLM regression suite for tracking quality across model upgrades: structures a versioned JSONL/CSV golden dataset, configures deterministic eval runs (temperature 0, seed), wires assertion layers (exact, semantic similarity, LLM-as-judge, rubric), enforces a pass-rate threshold with diff reporting vs the baseline model, and gates CI on regression. Use when upgrading an LLM provider model and needing a repeatable before/after quality gate, or when a prompt regression suite must track output quality across model versions over time.

openai-evals

Authors and runs OpenAI Evals - Python framework + registry for evaluating LLMs and LLM-backed systems with `oaieval {model} {eval-name}` CLI; supports template-based evals (Match / Includes / FuzzyMatch / ModelBasedClassify) defined in `evals/registry/evals/*.yaml` against JSONL data files in `evals/registry/data/`, plus custom Python eval classes implementing the Eval interface. Use when the user works with the openai/evals repo, needs the OpenAI-curated eval registry, or contributes new evals via PR to the registry.

ragas-evaluation

Authors and runs Ragas - RAG-pipeline evaluation framework with metrics organized into RAG (Faithfulness, Response Relevancy, Context Precision/Recall, Context Entities Recall, Noise Sensitivity), Natural Language Comparison (Factual Correctness, Semantic Similarity, BLEU/ROUGE/CHRF/Exact Match), Agents/Tool-Use (Topic Adherence, Tool Call Accuracy/F1, Agent Goal Accuracy), General Purpose (Aspect Critic, Rubrics-based Scoring), Nvidia (Answer Accuracy, Context Relevance, Response Groundedness), and Summarization. Use when the user evaluates a RAG pipeline (retriever + generator) and needs the deepest metric variety in the OSS LLM-eval space.