ARLoop: Fully-Open Pre-trained Looped Transformers with Attention Residuals

Researchers introduce ARLoop, a looped transformer architecture that uses attention residuals to selectively aggregate intermediate representations across model depth. Experiments across 140M, 370M, 0.6B, and 1.2B parameter scales demonstrate that sliding-window depth attention matches or outperforms full-history attention and yields strong improvements on math and coding tasks. Notably, a 1.2B ARLoop model pretrained on 0.5T tokens outperforms Huginn-3.5B and rivals similar-sized non-looped models despite using substantially less training data. Open article →