Learnings from Auditing Public Evaluations

Researchers at Prime Intellect audited several popular public AI evaluations—including AutomationBench, OfficeQA Pro v2, and HLE-Diamond—and identified design patterns, reward hacks, and brittle grading logic that can skew reported model capabilities. To make the evaluation landscape more robust, the team open-sourced fixes for these benchmarks, such as replacing prose request descriptions with JSON schemas and implementing synchronous monitors to detect web search leaks. Open article →