Programming Studyๆ—ฅๆœฌ่ชž

Disclosure: We earn a commission from Udemy when you buy a course through links on this page.

RAG Evaluation in Production: Trustworthy Agents

RAG Evaluation in Production: Trustworthy Agents

Udemy ยท English

About this course

Language
English
Category
Provider
Udemy
Data updated

AI-generated narration disclosure. The voice-over in this course is synthesized with a text-to-speech model from scripts written and reviewed by the instructor, and the on-screen material (notebooks, code, slides) is the instructor's own work. This is disclosed in line with Udemy's policy on AI-generated content. Wiring an LLM into a product takes an afternoon. Knowing whether it is right takes a discipline, and this course teaches that discipline end to end. The centre of the course is RAG evaluation: measuring retrieval and generation as separate failures, with Ragas, with LLM judges, and with test data you generate when you have none. Agent evaluation is the extension at the end โ€” once you can score a single answer you can score a tool-calling trajectory. "In production" is the other half of the title: tracing every call so a complaint becomes a reproducible case, turning a thumbs-down into a regression test, and putting cost and latency budgets beside quality so you can decide whether a change is worth shipping. We start with why evaluation is business-critical and the diagnostic questions every RAG owner should be able to answer. Then the tools: quantitative RAG evaluation with Ragas (context precision, faithfulness, answer relevancy) and how to read the per-sample table; LLM-as-a-judge with G-Eval, rubric design and judge bias; synthetic test data for the cold-start problem and how to validate it; input and output guardrails and how to know a guardrail is lying; observability and feedback loops with LangSmith. Two sessions go deeper than the usual cookbook: retrieval diagnostics (query-planning failures, temporal retrieval, multi-hop with and without planning, hit@k and MRR per failure class) and agent evaluation (tool-call correctness, trajectory scoring, judge rubrics, cost and latency budgets). The capstone audits a small customer-support RAG app with three planted defectsโ€ฆ

Courses in the same category