vLLM KV cache spill over PCIe with Llama 3.1 70B

pub.towardsai.net8 min read

When long-context GPU memory fills up during AI inference, the key-value cache must spill into CPU memory and SSD storage over PCIe. The author analyzes this production setup on an 8-GPU server running Llama 3.1 70B, examining cache sizes, the latency trade-offs between reloading from memory versus recomputing, and how traffic bursts turn PCIe into a queue. Open article →