Supercharging XLA-based inference: Announcing the launch of XLA Serving Engine

Google has open-sourced the XLA Serving Engine, a C++-based model export and inference runtime designed to enable a 'Train in Python, Serve in C++' workflow for the OpenXLA ecosystem. The self-contained engine allows developers to export ML models into an XLA SavedModel using StableHLO representation, removing Python interpreter overhead to achieve significantly higher throughput and lower latency. Already deployed internally across high-volume pipelines like YouTube Shorts and Google Ads, the runtime is now available to external hardware partners and Cloud TPU users. Open article →