Subquadratic Attention with a Hierarchical Tree-Structured Indexer | Zyphra

Researchers introduce Zyphra Hierarchical Attention (ZHA), a sparse attention method that builds and searches through a hierarchical tree of compressed keys to achieve subquadratic scaling for transformer models. According to the evaluation, ZHA can be adapted to existing pretrained models via a brief distillation phase, maintaining performance on challenging long-context reasoning and retrieval benchmarks while significantly outperforming dense attention and alternative sparse methods at ultra-long contexts. Open article →