ragas-evaluation

The model: assemble a dataset (question + answer + retrieval contexts + ground truth), import the metrics relevant to the evaluation goal, run evaluate(), and inspect per-metric per-row scores (per rg-gh (opens in new window)).

When to use

• The repo uses LangChain / LlamaIndex / Haystack / direct retriever→LLM RAG pipelines.
• You need RAG-specific retrieval + generation quality scoring, or agents-style eval (tool-call and goal accuracy), and want the widest metric variety in OSS LLM-eval.

For non-RAG prompt evals, prefer promptfoo-evaluation. For pytest-native LLM evals with a managed dashboard, prefer deepeval-evaluation.

Step 1 - Install

Per rg-gh (opens in new window):

pip install ragas

Or from source:

pip install git+https://github.com/explodinggradients/ragas

Step 2 - Custom metric quickstart

Per rg-gh (opens in new window) (verbatim):

import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures
key information.
Response: {response}
Answer with only 'accurate' or 'inaccurate'."""
)

# Score your application's output
async def main():
    score = await metric.ascore(
        llm=llm,
        response="The summary of the text is..."
    )
    print(f"Score: {score.value}")
    print(f"Reason: {score.reason}")

if __name__ == "__main__":
    asyncio.run(main())

DiscreteMetric is the pattern for custom rubric-based scoring; the built-in metrics in Step 3 follow a similar shape but are preconfigured.

Step 3 - Pick metrics

Ragas ships 30+ metrics across RAG, Natural Language Comparison, Agents/Tool-Use, SQL, General Purpose, Nvidia, and Summarization families. The RAG core: Faithfulness (claims grounded in retrieved context), Response Relevancy, Context Precision, and Context Recall. Pick 3 - 5 per pipeline.

Full per-family catalog with each metric's use: references/metrics.md, sourced from docs.ragas.io/en/stable/concepts/metrics/available_metrics/ (opens in new window).

Step 4 - Dataset shape

Ragas accepts a Hugging Face Dataset or pandas.DataFrame with columns matching the metrics being run:

See the per-metric pages on docs.ragas.io (opens in new window) for exact required-column lists.

Step 5 - Integration with retrieval frameworks

Ragas integrates with LangChain + LlamaIndex retrieval pipelines - the integration code captures contexts from the retriever and answer from the generator into the evaluation dataset automatically. Consult the per-framework integration docs on docs.ragas.io (opens in new window) when wiring; APIs evolve faster than this skill body and the canonical doc is the source of truth.

Step 6 - CI integration

Ragas does not ship a first-party CI action. Pattern: run evaluate() in a pytest fixture or a CLI script, compare per-metric scores against thresholds, fail CI on regression.

from ragas import evaluate
from ragas.metrics import faithfulness, answer_relevancy

result = evaluate(dataset, metrics=[faithfulness, answer_relevancy])
assert result["faithfulness"] >= 0.85
assert result["answer_relevancy"] >= 0.80

Anti-patterns

Limitations

• Many metrics require a judge LLM → cost scales with metric count × dataset size.
• Multimodal metrics need the multimodal extras (pip install ragas[multimodal]); check the per-metric doc on docs.ragas.io (opens in new window).
• API surface evolves - pin versions in requirements; the canonical doc is the source of truth (this skill body curates the mainstream patterns but does not re-litigate per-method signatures).

References

• rg-gh (opens in new window) - repository + install
• rg-metrics (opens in new window) - full metric catalog
• rg-docs (opens in new window) - full documentation including per-metric pages, integration guides
• deepeval-evaluation, promptfoo-evaluation - sister tools (different framework styles)