700 TPS on Kimi K3 with TPU Megakernels

Inferact has released a collection of megakernels for TPU v7 designed to improve decode throughput by more effectively utilizing on-chip VMEM and asynchronous prefetching. Their Kimi K3 implementation achieves over 700 tokens per second with speculative decoding, delivering roughly 1.4 to 2 times the throughput of a GB200 baseline at batch sizes one through eight without speculative decoding. Open article →