The MoE Training Handbook

This handbook guides engineers through training Mixture-of-Experts (MoE) models at scale, covering setup, hardware diagnostics, image building, tokenization, and a dense baseline. It details the added challenges of expert parallelism, routing, load balancing, and GPU-to-GPU token exchanges. The recipes rely on Slurm-managed clusters, PyTorch, and Megatron. Open article →