Everything I learned offloading MoE experts on 4x RTX 3090

The author shares lessons learned over four months of running large Mixture of Experts language models on a rig with four RTX 3090 GPUs and system RAM. The findings cover how offloading misses affect decode time, the importance of domain profiles and live caches over static sets, memory channel configurations, and the performance trade-offs of layer splits, tensor parallelism, and speculative decoding. Open article →