Recursive Sparse Attention for Million-Token Length Extrapolation

Researchers propose Recursive Sparse Attention (RecSA), a method that builds a tree over the KV cache to achieve million-token length extrapolation at logarithmic cost. Initialized from Qwen3-4B and trained on 0.62B tokens, RecSA outperforms dense baselines on RULER across lengths from 131K to 2M while accelerating end-to-end prefill and decoding. Open article →