qa-llm-evaluation
LLM and prompt evaluation with Promptfoo, DeepEval and Ragas, Giskard vulnerability scans, Langfuse tracing, and golden-dataset regression suites. Audits eval suites for methodology errors. Agents (prompt-eval-reviewer, llm-red-team-planner) review an eval suite and plan adversarial probe campaigns across the OWASP LLM Top 10.
/plugin install qa-llm-evaluation@testland-qaFirst time? Add the marketplace with /plugin marketplace add testland/qa
- Category
- AI & ML
- Version
v1.4.4- Contents
- 7 skills · 2 agents
- Part of
- qa-role-ai
Skills7
deepeval-evaluationUse when LLM or RAG evaluation should run as pytest tests: DeepEval LLMTestCase, built-in metrics such as G-Eval, answer relevancy, faithfulness, contextual recall and precision, hallucination, bias and toxicity, deepeval test run with assert_test, and the Confident AI dashboard.skillgiskard-llmUse when an LLM app needs adversarial or red-team tests on top of functional evals.skilllangfuse-tracingUse when an LLM app needs production tracing, monitoring or debugging of model calls, or production traces should feed offline eval datasets.skillllm-eval-anti-patternsUse when an LLM eval suite already gates a release, model swap or prompt change and nobody has audited how it was built.skillllm-regression-suite-authorUse when upgrading an LLM model or provider needs a repeatable before/after quality gate, or prompt quality must be tracked across model versions.skillpromptfoo-evaluationUse when LLM prompts or RAG pipelines need eval-driven regression tests with Promptfoo: promptfooconfig.yaml providers, prompts and tests, deterministic, semantic, model-graded, latency and cost assertions, promptfoo eval and view, and CI gating.skillragas-evaluationUse when a RAG pipeline (retriever plus generator) or an agent needs the widest set of evaluation metrics: Ragas faithfulness, response relevancy, context precision and recall, noise sensitivity, factual correctness, tool-call accuracy, agent goal accuracy and rubric scoring.skill
Agents2
llm-red-team-plannerAction-taking orchestrator that plans and scaffolds a multi-class LLM adversarial probe campaign beyond canned scanners - enumerates an attack taxonomy (jailbreaks, indirect prompt injection chains, data exfiltration, harmful-content bypass, OWASP LLM Top 10 classes), maps each class to a Giskard detector or a promptfoo red-team plugin, sequences the campaign into phases, and writes the resulting scan scripts and promptfoo redteam YAML configs.agentprompt-eval-reviewerAdversarial reviewer for an LLM eval suite (Promptfoo, OpenAI Evals, DeepEval, Ragas, Giskard, Langfuse-driven, or custom).agent
About qa-llm-evaluation
LLM and prompt evaluation. Six per-tool skill wrappers covering the mainstream OSS LLM-eval ecosystem (Promptfoo + OpenAI Evals + DeepEval + Ragas for functional eval; Giskard for adversarial scan; Langfuse for production observability + offline-eval feedback loop) plus an adversarial reviewer agent that flags 8 anti-patterns across any of these frameworks.