Meta releases Cap-and-Track LLM dispatch design
Researchers present Cap-and-Track, an asynchronous inference design implemented by Meta's traffic shaper to dispatch LLM requests across provider capacity and internal GPUs using only refusal signals. The system serves a classifier on 43% fewer H100s than its queued, fixed-rate precursor while following capacity ceilings without output-length prediction or retuning. Open article →