Sebastian Raschka discusses constant LR pretraining

sebastianraschka.com1 min read

The author analyzes an emerging pre-training trend in large language models that replaces traditional learning rate decay phases with constant learning rates combined with checkpoint averaging. This approach may simplify the training process by eliminating the need to commit to a total training length upfront. The post categorizes various models according to whether they use classic cosine decay, Warmup-Stable-Decay, or no decay recipes. Open article →