LaunchDarkly proposes pass^k for AI agent consistency

The author argues that measuring AI agent reliability requires tracking consistency across multiple runs using metrics like pass^k and flip rate, rather than relying solely on single-run accuracy or low temperatures. To catch regressions before release, teams should run candidate models multiple times against labeled datasets, classify outcomes row by row, and block rollouts if any critical decisions lose stability. Open article →