Programming Studyๆ—ฅๆœฌ่ชž

Disclosure: We earn a commission from Udemy when you buy a course through links on this page.

Ship Reliable AI Agents: LLM Evals, CI Gates and Monitoring

Ship Reliable AI Agents: LLM Evals, CI Gates and Monitoring

Udemy ยท English

About this course

Language
English
Category
Provider
Udemy
Data updated

Your agent's pull request is green. Every unit test passes. That afternoon, a customer asks about a refund, and the bot explains a refund policy your company doesn't have. Unit tests check that your code runs. They don't check that an answer is right. This course teaches the loop that does: traces, error analysis, a labelled dataset, graders, a CI gate, and monitoring of live traffic, built in plain Python and pytest on a small support agent you run on your own machine. How it worksWatch. Each lecture is short and animated: traces unfold span by span, failure categories take shape from your notes, thresholds are drawn against their error barsRun it. You get the companion repository as a zip with its full git history, one starting branch per section. Everything runs in replay mode against recorded responses, so you need no API key and spend nothing on tokens. Practise. Every section ends with a short quiz and a practice exercise in the repo, with a check command that tells you when you've got it right. What you'll work onTraces: instrument an agent with OpenTelemetry so every run leaves a record of its model calls, tool calls and retrievals, and keep user data out of them. Error analysis: read traces, write notes, group them into a failure taxonomy and turn it into a pass or fail labelled dataset. Code checks first: write pytest checks with passing and failing examples before reaching for an LLM judge.LLM judges you can trust: write a binary judge, validate it against human labels with TPR, TNR and Cohen's kappa, and measure what each run costs.RAG: score retrieval (precision, recall, MRR) and generatioโ€ฆ

Courses in the same category