Blackwell to Rubin boosts low-latency inference capacity

The author analyzes how modern hardware improvements—from Blackwell to Rubin and beyond—disproportionately boost AI inference performance at the interactivity level compared to basic concurrency metrics. By modeling a 3-trillion-parameter extrapolation of Llama 3, the analysis shows that faster chips shrink the fixed cost of every decode step, significantly increasing the user capacity under low-latency targets. However, due to ongoing hardware supply constraints, the industry will likely continue running inference on older hardware to meet infinite demand for tokens. Open article →