Researchers propose GPU operating system to boost AI throughput

Researchers argue that GPUs sit mostly idle while serving AI models because steps wait on memory, and propose that a GPU operating system is needed to schedule urgent work and fill idle capacity. They report that their research system runs a whole model faster than vLLM while sharing the GPU at under a two percent slowdown, and specify the compilers and chip features needed for broader scaling. Open article →