Stanford and Google DeepMind Introduce TURTLE for Evolving Harder Tool-Use Benchmarks

Researchers from Stanford and Google DeepMind have introduced TURTLE, a method that evolves existing static tool-use benchmarks using AlphaEvolve to create harder, still-solvable tests. By mutating tasks with filters, extra steps, and hidden lookups, the approach targets a 50 percent solver success rate to prevent models from easily acing static evaluations. Testing showed that evolved tasks successfully challenged models that previously solved the original benchmarks with ease. Open article →