getphase

Phase 6 · 4–6 weeks

Pre-Training — Scaling, Data, Distributed Systems

Estimate compute-optimal trade-offs; use FSDP and parallelism that doesn't OOM at 3am.

What this phase teaches

Kaplan vs. Chinchilla scaling laws; the ~20-tokens-per-parameter rule. Data curation (FineWeb, FineWeb-Edu, RedPajama v2, Dolma); MinHash dedup; quality filters. ZeRO 1/2/3, PyTorch FSDP/FSDP2, DeepSpeed, Megatron-LM. Expert / tensor / pipeline / sequence parallelism. FlashAttention v1/v2/v3. FP8 training.

Anchor resources

  • Stanford CS336 Lectures 7–9, 11, 13–14
  • Hugging Face’s free Ultra-Scale Playbook
  • Papers: 2001.08361 (Kaplan), 2203.15556 (Chinchilla), 2205.14135 (FlashAttention)