Researchers redesign Olmo-core for scalable MoE training
Researchers present a redesigned training stack in Olmo-core to achieve efficient and scalable Mixture-of-Experts training by replacing Fully Sharded Data Parallel with Distributed Data Parallel, Expert Parallelism, and Pipeline Parallelism. The new architecture introduces NVSHMEM-based rowwise expert parallelism, synchronization-free routing, and closely integrated MXFP8 to reduce compute and transport costs. The system supports training configurations from 12.9 billion up to 1.2 trillion parameters on up to 512 GPUs. Open article →