Physics of Language Models: Canon Layers at Scale and Synthetic Pretraining

arxiv.org1 min read

Researchers studied 64 from-scratch pretraining runs of Transformers and linear models to evaluate architectural differences at scale. They found that at 1-2T tokens, local token mixing layers consistently improve Transformers and allow simpler linear models to match more complex architectures like Gated DeltaNet and Mamba2. However, both Transformer and linear models struggled with basic in-context retrieval and two-hop reasoning tasks at an 8B/1T scale. Open article →