Agent Substrate autoscaler increases latency with local Ollama

The author argues that scaling agent workers against a backend with a serial generation queue, such as a local Ollama instance, increases latency and forms a larger waiting room instead of improving throughput. When autoscalers use slot demand or busy-plus-queued metrics without accounting for model concurrency, adding workers merely increases occupancy cosmetics. To fix this, the author recommends capping the useful worker pool at the backend generation concurrency limit rather than relying on capacity scaling alone. Open article →