Ramp launches SWE Bench V2 with LLM judges

builders.ramp.com6 min read

Ramp has launched SWE Bench V2, an updated coding benchmark that replaces traditional test suites with LLM judges to evaluate AI coding models. By separating the identification of code differences from assessing their materiality, and introducing a multi-model deliberation pipeline, the company achieved significantly higher agreement among judges and strong alignment with human engineering reviews. Open article →