Red Hat benchmarks KV cache offloading in OpenShift AI 3.5 on H100 GPUs

Red Hat evaluates tiered KV cache offloading on NVIDIA H100 GPUs within OpenShift AI 3.5, using CPU DRAM, local NVMe, and CephFS to reduce inference recomputation. Testing shows that secondary storage tiers such as NVMe or CephFS significantly improve throughput and reduce time to first token for long-context workloads when working sets exceed CPU capacity. The authors recommend a workload-driven evaluation process combining FIO storage benchmarking with crossover calculations to select the appropriate deployment topology. Open article →