Researchers introduce HiSA to sparsify RoPE attention
Researchers propose Hidden Sparse Attention (HiSA), a method that computes partial dot products in the hidden feature dimension of RoPE attention based on token positions. Training large language models from scratch with HiSA improves performance while computing only about a third of the query-key products and speeding up inference. The authors also extended HiSA to FlashAttention-3 kernels and released code on GitHub. Open article →