Phase 4 · 3–4 weeks
Transformers Deep Dive
Implement multi-head attention and a full decoder-only Transformer from a blank file.
What this phase teaches
Scaled dot-product attention, multi-head and grouped-query attention (GQA), masking (causal, padding), positional encodings (sinusoidal / learned / RoPE / ALiBi), pre-norm vs post-norm, encoder-only (BERT), decoder-only (GPT), encoder-decoder (T5/BART).
Anchor resources
- Karpathy — “Let’s build GPT: from scratch, in code, spelled out” and “Let’s reproduce GPT-2 (124M)” (
karpathy/build-nanogpt) - Sebastian Raschka — Build a Large Language Model (From Scratch)
- Hands-On Large Language Models (Alammar & Grootendorst)
- Must-read papers: “Attention Is All You Need” (1706.03762), BERT (1810.04805), GPT-3 (2005.14165), T5 (1910.10683), RoPE (2104.09864)