Researchers introduce SHADOWPREFILL for LLM routing cache reuse

Researchers introduce SHADOWPREFILL, a system designed to maintain prefix-cache reuse during per-step model routing in agentic LLM serving. By prefilling a session's context on unchosen models using idle instance capacity, SHADOWPREFILL avoids the latency and cost penalties associated with cold model switches. Experiments across multiple benchmark suites demonstrate substantial reductions in switch-call latency, overall LLM-call latency, and routed token costs. Open article →