Author criticizes Baseten metric cost per merged PR

The author argues that measuring coding agent performance by cost per successful task creates severe survivorship bias and rewards cheap models that quit early. Instead, the author advocates for evaluating agent quality through head-to-head performance matrices, Elo ratings, and task slicing by workload shape. Open article →