← CurriculumWeek 08 / 10 · Quality
08

Evals: Knowing Your AI Actually Works

Stop shipping vibes. Start shipping measured outputs. This is the single biggest discipline that separates client work from playing around.

Topics covered

  1. 01Why evals matter — especially for client-facing AI work
  2. 02Types of evals: deterministic, LLM-as-judge, human review
  3. 03Writing test cases and building a golden dataset
  4. 04Iterating prompts and systems using eval feedback (the inner loop)
  5. 05Eval tooling: Braintrust, LangSmith, or a simple Python harness
  6. 06Wiring evals into client deliverables and acceptance criteria
  7. 07Eval anti-patterns: cherry-picking, judge-model bias, vanity metrics

Build challenge

Deliverable

Eval suite (5–10 cases) for the Week 7 agent. Iterate the agent based on results. Document the deltas.