Labs increasingly able to pretrain strong models

The author argues that pretraining a strong model is becoming increasingly feasible due to a growing and improving pool of high-quality synthetic tokens on the web. As frontier labs serve tokens at scale to pay back capital expenditures, their competitive edge slowly leaks into the public domain and commoditizes their models. This dynamic may explain why labs choose to keep their best models private. Open article →