Ragas

What it is

Ragas is a metric suite for evaluating the quality of retrieval and answer faithfulness in RAG systems.

It provides measurable scores, turning subjective assessments into quantitative metrics.

Its synthetic test-set generation capability is notable, as it allows for regression suite creation even without labelled data.

Where it fits in a deployment stack

Tools like Ragas are used to assess the performance of language models before they are deployed to production.

In a production system, they normally sit alongside the model training and deployment pipelines, providing critical metrics for go-live reviews.

They take over the task of evaluating model quality, freeing up developers to focus on model improvement and deployment.

How to evaluate it

To evaluate Ragas, you need to consider the specific metrics it provides and how they align with your model's requirements.

• Feed it a dataset with known retrieval and faithfulness issues to inspect its diagnostic capabilities.
• Measure p95 latency of the retrieval step under various concurrent session loads.
• Inspect the output of its synthetic test-set generation to verify its quality and relevance.
• Compare its metrics to those from other evaluation methods to assess its accuracy.
• Verify the correctness of its faithfulness scores through manual inspection of a subset of results.
• Check the computational resources required to run Ragas and its impact on overall system performance.

What to verify before you commit

Before committing to Ragas, you should verify several critical aspects of its functionality, licensing, and operational requirements.

• Check the licence terms to ensure they align with your project's requirements and constraints.
• Verify how Ragas handles sensitive data and whether it meets your organization's data protection standards.
• Assess the operational cost of running Ragas, including any computational resources or personnel required.
• Test its failure behaviour to understand how it responds to errors or unexpected input.
• Evaluate the potential for vendor lock-in and the ease of migrating to alternative solutions if needed.
• Confirm that Ragas can integrate with your existing model training and deployment pipelines.