Phase 12 · 3–4 weeks
Deployment & Inference Optimization
Serve at >1000 tok/s/GPU. Quantize down to consumer hardware without losing your benchmarks.
What this phase teaches
KV cache footprint; PagedAttention and vLLM; continuous batching; chunked prefill; prefix caching; speculative decoding (Medusa, EAGLE, lookahead); quantization (GPTQ, AWQ, GGUF, NF4, FP8, FP4, MXFP4, AQLM); serving stacks (vLLM, SGLang, TGI, TensorRT-LLM, llama.cpp, MLC-LLM, Ollama); structured generation (XGrammar, Outlines); disaggregated prefill/decode.
Anchor resources
- CS336 Lecture 10 (Inference)
- Papers: 2309.06180 (vLLM/PagedAttention), 2210.17323 (GPTQ), 2306.00978 (AWQ), 2401.10774 (Medusa)