Open-Source AI Models: Total Parameters Explode While Active Parameters Lag

The author observes that while total parameters in open-source AI models are growing rapidly, active parameters are not, driven by Mixture of Experts (MoE) architectures. Although MoE reduces per-token compute costs, it creates severe memory and networking bottlenecks as every expert must reside in memory and route tokens across GPUs. Consequently, the author argues that infrastructure constraints are shifting from FLOPs to high-bandwidth memory capacity, interconnect, and power density. Open article →