Engineer details Kimi K3 trillion-parameter cluster on GB300 NVL72
An engineer at a GPU cloud company explains the capacity planning, memory costs, and routing considerations required to deploy the trillion-parameter Kimi K3 language model for 100,000 concurrent users. The practical deployment utilized 21 racks of GB300 NVL72 hardware to handle 2.07 million tokens per second while maintaining latency goals for 99 percent of requests. The author breaks down memory consumption factors such as KDA state and MLA latents for a single chat request. Open article →