Zyphra scales ZHA attention to Qwen3.8 2.4T
Researchers at Zyphra scaled their Zyphra Hierarchical Attention (ZHA) method to frontier open-weight models including Qwen, Nemotron, MiMo, and GPT-OSS. They found that ZHA conversion preserves overall model performance and requires inexpensive local distillation, but that computational scaling is limited in some models by the structure of their underlying attention distributions, such as diffuse early layers or long-context attention flattening. Open article →