KV cache remains transformer decode bottleneck

stateofthefuture.io4 min read

The author argues that the KV cache remains an irreducible memory-bandwidth constraint in transformer inference that cannot be eliminated by weight-focused HBM-free hardware architectures. Unlike model weights, the KV cache grows dynamically per user and per token, causing memory traffic to scale linearly with batch size and context length. Consequently, alternative hardware approaches are best suited for small-model, short-context, latency-bound workloads rather than frontier large language model decode. Open article →