Introducing FrontierHarness Eval v1.1 — RUNTA

runta.com14 min read

Researchers expanded their FrontierHarness benchmark to 22 harnesses across 5 models, finding that developers should consider switching harnesses when changing models to maximize performance and cost-efficiency. The study revealed that the model accounts for the vast majority of cost variance, while the choice of harness can still vary costs by up to five times for the same model. Furthermore, higher cost did not correlate with higher pass rates, and the majority of expenses came from failed runs that ended in confident final answers. Open article →

More on this topic