KV cache remains transformer decode bottleneck
The author argues that the KV cache remains an irreducible memory-bandwidth constraint in transformer inference that cannot be eliminated by weight-focused HBM-free hardware architectures. Unlike model weights, the KV cache grows dynamically per user and per token, causing memory traffic to scale linearly with batch size and context length. Consequently, alternative hardware approaches are best suited for small-model, short-context, latency-bound workloads rather than frontier large language model decode. Open article →