Qwen3.5–35B-A3B MoE parallelism explained on 8 GPUs
The author examines how data, tensor, pipeline, and expert parallelism combine in large language models, providing a detailed case study of serving a 35-billion-parameter mixture-of-experts model on eight GPUs. The analysis demonstrates how expert parallelism reuses tensor and data parallelism groups to change memory distribution and GEMM shapes without reducing total expert weights per GPU. The author explains that combining expert parallelism with DP-attention reduces KV cache duplication and enables larger batches for better throughput. Open article →