EUROPA scales 1T MoE pretraining on JUPITER with PyTorch and TorchTitan

The EUROPA consortium successfully scaled the pretraining of a roughly 1-trillion-parameter Mixture-of-Experts model on public JUPITER supercomputer infrastructure using PyTorch and TorchTitan. By implementing optimizations such as FlexShard placement for the Muon optimizer, activation-checkpoint offloading to the Grace CPU, and asynchronous all-gather setups, the team boosted throughput from 90 to approximately 1,300 tokens per second per GPU across 384 NVIDIA GH200 GPUs. Open article →