
RAG Evaluation in Production: Trustworthy Agents
Udemy ยท English
About this course
- Language
- English
- Category
- Provider
- Udemy
- Data updated
AI-generated narration disclosure. The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor's own work. This is disclosed in line with Udemy's policy on AI-generated content. Wiring an LLM into a product takes an afternoon. Knowing whether it is right takes a discipline, and this course teaches that discipline end to end. The centre of the course is RAG evaluation: measuring retrieval and generation as separate failures, with Ragas, with LLM judges, and with test data you generate when you have none. Agent evaluation is the extension at the end โ once you can score a single answer you can score a tool-calling trajectory. "In production" is the other half of the title: tracing every call so a complaint becomes a reproducible case, turning a thumbs-down into a regression test, and putting cost and latency budgets beside quality so you can decide whether a change is worth shipping. We start with why evaluation is business-critical and the diagnostic questions every RAG owner should be able to answer. Then the tools: quantitative RAG evaluation with Ragas (context precision, faithfulness, answer relevancy) and how to read the per-sample table; LLM-as-a-judge with G-Eval, rubric design and judge bias; synthetic test data for the cold-start problem and how to validate it; input and output guardrails and how to know a guardrail is lying; observability and feedback loops with LangSmith. Two sessions go deeper than the usual cookbook: retrieval diagnostics (query-planning failures, temporal retrieval, multi-hop with and without planning, hit@k and MRR per failure class) and agent evaluation (tool-call correctness, trajectory scoring, judge rubrics, cost and latency budgets). The capstone audits a small customer-support RAG app with three planted defectsโฆ











