Key Takeaways
- RAG evaluation requires measuring both retrieval and generation: a system can retrieve perfectly but generate poorly, or vice versa. Evaluate each stage independently.
- Faithfulness is the most critical metric: an answer that sounds correct but contains hallucinated information is worse than no answer at all.
- LLM-as-judge automates evaluation at scale, using a separate model to score correctness, faithfulness, and relevance against ground truth.
- 50-100 evaluation pairs are sufficient to detect most quality issues. Expand the dataset as you discover new failure modes.
- RAGAS and DeepEval frameworks automate the evaluation pipeline, enabling continuous quality monitoring on every system change.
RAG Evaluation Metric Framework
| Metric | What It Measures | Target Range | Failure Indicates | Measurement Method |
|---|---|---|---|---|
| Answer correctness | Match with ground truth | >85% | Poor generation or retrieval | LLM-as-judge or F1 score |
| Faithfulness | Grounding in retrieved context | >90% | Hallucination problem | NLI model or RAGAS |
| Context relevance | Quality of retrieved chunks | >80% | Retrieval or chunking issue | Human annotation |
| Context recall | Coverage of required information | >75% | Missing knowledge or bad embeddings | Ground truth comparison |
| Answer relevance | Response addresses the question | >90% | Prompt engineering issue | LLM-as-judge |
| Latency (p95) | End-to-end response time | <3 seconds | Infrastructure or architecture issue | Application monitoring |
Metric Priority
If you can only track two metrics, choose faithfulness and answer correctness. Faithfulness catches hallucinations (the most dangerous failure), and correctness catches wrong answers. Other metrics help diagnose root causes but these two capture the user experience.
Building Effective Evaluation Datasets
| Query Type | Count (minimum) | Purpose | Example |
|---|---|---|---|
| Factual | 20 | Test precise information retrieval | "What is the return policy for opened items?" |
| Explanatory | 10 | Test reasoning and synthesis | "How does the pricing tier work?" |
| Comparison | 10 | Test multi-document retrieval | "What is the difference between Pro and Enterprise?" |
| Negative | 5 | Test boundary handling | "What is the weather today?" (out of scope) |
| Multi-hop | 5 | Test complex reasoning chains | "If I am on the Pro plan and upgrade, what changes?" |
| Ambiguous | 5 | Test clarification behavior | "How much does it cost?" (no product specified) |
- Annotate ground truth: for each question, write the correct answer and list which document chunks contain the supporting information
- Include edge cases: questions about topics at chunk boundaries, questions requiring information from multiple documents, and questions the knowledge base cannot answer
- Update regularly: add new evaluation pairs when you discover production failures or when the knowledge base is updated significantly
- Version the dataset: track dataset changes alongside system changes so you can attribute metric changes to the correct cause
Automated Evaluation Frameworks
| Framework | Key Features | Metrics Supported | Integration |
|---|---|---|---|
| RAGAS | LLM-based evaluation, widely adopted | Faithfulness, relevance, correctness | LangChain, LlamaIndex |
| DeepEval | 14+ metrics, CI/CD integration | All RAGAS + custom metrics | Pytest, GitHub Actions |
| LangSmith | Tracing + evaluation by LangChain team | Custom evaluators, tracing | LangChain native |
| Weights & Biases | Experiment tracking + evaluation | Custom metrics, comparison | Framework-agnostic |
| Custom (LLM-as-judge) | Full control over evaluation criteria | Any metric you define | Any framework |
Start with RAGAS
RAGAS provides the fastest path to automated RAG evaluation. Install the library, point it at your evaluation dataset, and get faithfulness, context relevance, and answer correctness scores in minutes. It uses LLM-based evaluation, so it requires an API key for the judge model.
Continuous Improvement Process
- Run evaluation on every change: automate evaluation in your CI/CD pipeline. Any change to chunking, embeddings, prompts, or retrieval should trigger an eval run.
- Analyze failures by category: group evaluation failures into retrieval failures (wrong chunks), generation failures (hallucination), and relevance failures (correct info, wrong answer). Each category has different fixes.
- Fix retrieval first: most RAG quality issues stem from poor retrieval. If the right chunks are not retrieved, no amount of prompt engineering fixes the answer. Improve chunking, embeddings, or search strategy.
- Track metrics over time: plot evaluation scores weekly to detect gradual degradation. Knowledge base staleness, embedding model updates, and content changes can cause slow quality drift.
- Production monitoring: log user queries, retrieved chunks, generated answers, and user feedback (thumbs up/down). Analyze weekly for new failure patterns not captured in the evaluation dataset.
Frequently Asked Questions
Related services from Verbsz Technologies
If this article is relevant to a project on your roadmap, these are the services our team typically delivers.
AIRAGEvaluationQualityMetrics