Allen AI OLMoE-1B-7B routing traced on NVIDIA A100

Mixture-of-Experts (MoE) models decouple parameter storage in GPU memory from active floating-point math per token, allowing models like Alibaba's Qwen3.5-35B-A3B to store 35 billion parameters while executing only 3 billion per token. The author traces token routing across depth using Allen AI's OLMoE-1B-7B model on an NVIDIA A100 GPU and finds that a single expert often dominates the entrance layers across different tasks. Additionally, experiments show that cutting per-token math substantially yields only a modest speedup at batch size 1. Open article →