FlashAttention reduces attention memory but not KV cache

Flash Attention optimizes attention calculation by using tiles to compute results without materializing the full N by N score matrix in GPU memory, reducing workspace memory and data movement. However, the author explains that it does not reduce model weights or the persistent KV cache, which continues to grow proportionally with the context length. Addressing large context limits requires combining Flash Attention with techniques such as grouped-query attention, PagedAttention, or KV-cache quantization. Open article →