getphase

Phase 5 · 3–4 weeks

Modern LLM Architectures & MoE

Llama 4, Qwen3, DeepSeek V3/R1 — what's actually inside the frontier.

What this phase teaches

The Llama 1→4 lineage, Mistral/Mixtral, Qwen 2.5/Qwen3, DeepSeek V2/V3/R1, Gemma, Phi. MoE routing (top-k, switch, shared experts, expert parallelism), SwiGLU, RMSNorm, GQA + Multi-head Latent Attention (DeepSeek MLA), iRoPE for long context, FP8 training.

Anchor resources

  • Stanford CS336 Lectures 3 (Architectures), 4 (MoE), 5 (GPUs), 6 (Kernels)
  • Sebastian Raschka’s architecture-comparison hub
  • Papers: 2407.21783 (Llama 3), 2505.09388 (Qwen3), 2412.19437 (DeepSeek-V3), 2501.12948 (DeepSeek-R1), 2101.03961 (Switch Transformer), 2401.04088 (Mixtral)