Evaluating llama.cpp's MoE cache performance on an RTX 5090

An evaluation of llama.cpp's new MoE cache on a 122B model running on an RTX 5090 found that it increased decode speed from 42.5 to 68.7 tokens per second and reduced repository fixing time from 121 to 85 seconds. The cache increased VRAM usage from 28,850 to 31,752 MiB and increased power draw by about 75 W. Open article →