Ramp launches SWE Bench V2 with LLM judges
Ramp has launched SWE Bench V2, an updated coding benchmark that replaces traditional test suites with LLM judges to evaluate AI coding models. By separating the identification of code differences from assessing their materiality, and introducing a multi-model deliberation pipeline, the company achieved significantly higher agreement among judges and strong alignment with human engineering reviews. Open article →