Apple releases Krites LLM-judged semantic caching policy
Researchers introduce Krites, an asynchronous, LLM-judged caching policy designed for tiered large language model architectures to expand static cache coverage without altering critical path serving decisions. When a prompt falls just below the static similarity threshold, Krites uses an LLM judge to verify if the static response is acceptable and promotes approved matches into the dynamic cache. Trace-driven simulations show that Krites increases the fraction of requests served with curated static answers by up to 3.9 times for conversational and search workloads while maintaining unchanged critical path latency. Open article →