Spotify explains limits of shipping on offline evals

engineering.atspotify.com10 min read

Building AI and LLM features often relies on offline evaluations to iterate quickly without slow A/B tests, but relying on them exclusively requires strict assumptions about data homogeneity and measurement error. The author explains that naive estimators can severely underestimate the correlation between offline and online results due to noise, and that arm-dependent bias in LLM judges can distort treatment effects. Ultimately, teams must maintain clean historical datasets, account for noise, and continuously use audit pools to detect drift if they wish to safely ship based on offline evaluations alone. Open article →