I ran 120B MoE models on my M1 Max

An engineer developed a prototype inference engine to run large mixture-of-experts language models on an Apple M1 Max with 32GB of RAM by keeping most weights on the SSD and enforcing a 16 GiB process-memory budget. In benchmarks against TUFF 10.2.2 using matching weights on gpt-oss-120b, the custom engine achieved about 36 percent higher token generation speed while significantly reducing file cache usage and system memory pressure. Open article →