getphase

Phase 7 · 4–5 weeks

Fine-Tuning & Alignment

Take a base model to instruction-following with the right algorithm for your budget.

What this phase teaches

SFT, instruction tuning, chat templates. PEFT (LoRA, QLoRA, DoRA, IA³). RLHF (reward model + PPO). DPO, KTO, ORPO, SimPO, IPO. GRPO + RLVR (DeepSeek-R1 style). Constitutional AI / RLAIF. Reward hacking and mitigation.

Anchor resources

  • HF LLM Course chapters 11–12
  • Sebastian Raschka — Build a Reasoning Model (From Scratch) (2025)
  • Maxime Labonne’s LLM Course — github.com/mlabonne/llm-course
  • Papers: 2106.09685 (LoRA), 2305.14314 (QLoRA), 2203.02155 (InstructGPT), 2305.18290 (DPO), 2402.03300 (GRPO)