Alibaba Qwen4Exp Architecture Revealed Through Kernel PRs

Recent kernel pull requests across multiple frameworks reveal the internal architecture of Alibaba's Qwen4Exp model, which combines Gated DeltaNet linear layers and Qwen Sparse Attention with parallel Hyper-Connection streams. Fusing a Qwen4Exp decoder layer into a persistent kernel on an AMD MI355X GPU increased output speed by 37.4%. The model pairs this active compute with a large 102.4 GB frozen Per-Layer Embedding n-gram lookup table. Open article →