Evolving Infra: Agent-Adaptive LLM Serving at Scale

Researchers present Evolving Infra, an agent-adaptive serving framework designed to optimize LLM inference for agentic workloads by decoupling compute allocation from key-value cache management. The system allows nodes to dynamically switch between prefill and decode phases while preserving cached agent context through branch-aware routing and eviction. Evaluations across various coding, math, and computer-use tasks show significant reductions in mean end-to-end request latency and P95 time-to-first-token compared to production frameworks. Open article →