Phase 11 · 2–3 weeks
Evaluation
Trustworthy evals before you trust any output. Goodhart's law in practice.
What this phase teaches
Capability benchmarks (MMLU, GPQA, HumanEval, MBPP, LiveCodeBench, SWE-bench, MATH-500, AIME, ARC-AGI, BIG-bench Hard, HellaSwag, TruthfulQA, IFEval), open-ended evals (MT-Bench, Chatbot Arena, Arena-Hard), LLM-as-a-judge failure modes (position bias, verbosity, self-preference), RAG evals (RAGAS, TruLens), contamination detection.
Anchor resources
- EleutherAI
lm-evaluation-harness - Hugging Face
lighteval,ragas - UK AISI
inspect_ai - W&B Weave,
promptfoo - Papers: 2009.03300 (MMLU), 2310.06770 (SWE-bench), 2306.05685 (MT-Bench), 2403.04132 (Chatbot Arena)