Hugging Face Transformers enables local 180B MoE expert streaming

huggingface.co9 min read

The author implemented expert streaming for Mixture of Experts models within Hugging Face Transformers, enabling a 180-billion parameter model to run locally on consumer hardware. By using a slot pool to map routed expert IDs to preallocated device tensors, the approach reuses existing grouped matrix multiplication kernels without requiring custom C++ extensions. Benchmarks on a 32GB Mac M5 show the implementation achieves within six percent of the Strata inference engine on short decodes. Open article →