Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

Researchers introduced Latent Sparse Attention (LSA), a method to achieve subquadratic ultra-long context attention by hierarchically clustering keys in a learned latent space. The approach includes two variants: Distilled-LSA, applied to pre-trained models via distillation, and Native-LSA, trained natively on next-token loss. Experiments show significant speedups in prefill and decode stages while maintaining accuracy across tasks compared to full and traditional sparse attention methods. Open article →