Superfluid open-source local LLM server released for distributed inference
Researchers introduce Superfluid, an open-source local LLM server designed to handle inference across single devices and distributed heterogeneous hardware fleets. The system schedules interactive requests ahead of agent work, caches shared prompts, and pools machines of different chip vendors into a single API endpoint. Evaluations show significant improvements in time-to-first-token, throughput, and fault tolerance compared to existing local servers. Open article →