Judgment Labs evaluates Agent-as-a-Judge for 1M+ token trajectories

judgmentlabs.ai12 min read

Researchers at Judgment Labs evaluated methods for judging long-horizon AI agent trajectories exceeding one million tokens and found that an Agent-as-a-Judge approach significantly outperforms traditional LLM-as-a-Judge and chunk-screening techniques. By searching trajectories iteratively using SQL and code execution instead of reading the entire run, the Agent-as-a-Judge achieved higher evidence F1 scores at lower costs, particularly past one million tokens where context rot degrades linear judges. Open article →