HiCache: Across Parallelism & Distributed Serving for Agentic Workloads

HiCache extends radix cache prefix reuse across host memory, external storage, parallelism, speculative decoding, and prefill-decode disaggregation to improve production LLM serving. By implementing context parallelism page-based KV sharding and pipeline synchronization, the system increases cache hit rates and reduces prefill times for long-context workloads. Open article →