Inference Performance Index: changing the way we measure inference capacity

Epoch AI has introduced the Inference Performance Index (IPI) to measure AI chip inference capacity by evaluating how many simultaneous requests a chip can serve across a range of required output speeds. Version 1 focuses on agentic workloads and expresses scores in Nvidia GB300 equivalents. The authors find that memory and inter-chip communication can matter more than arithmetic throughput, meaning raw FLOP/s comparisons can significantly underestimate inference capacity. Open article →