Agent Substrate rejects CPU autoscaling

The author argues that CPU utilization is an ineffective signal for autoscaling agent workers because LLM sessions spend most of their time waiting on I/O, resulting in single-digit mCPU usage even when the pool is fully saturated. Instead, platform teams should autoscale based on sustained client rejections and queue occupancy. A practical autoscaling policy that responds quickly to queue depth while sizing down based on occupancy is proposed as an alternative. Open article →