GlobeBench: Evaluating Large Language Models as Simulation Environments

Researchers have introduced GlobeBench, a benchmark comprising 1,632 tasks across five domains to evaluate whether language models can accurately predict environment responses in agentic workflows. Among 20 tested models, GPT-6 Astra achieved the highest success rate at 49.57%, while no model solved half the benchmark and 631 tasks remained completely unsolved. The creators have open-sourced the entire benchmark, code, and model answers on GitHub. Open article →