BenchProofer: Formally verified coding benchmarks, on demand - Amazon Science
Amazon has released BenchProofer, an open-source tool that uses automated reasoning and formal verification to strengthen coding benchmarks. Alongside the tool, Amazon released SWE-Proof, adding formal specifications, implementations, and proofs to all 500 tasks in the SWE-Bench Verified dataset. Evaluations showed that agents achieve higher verified pass rates when provided with formal specifications, though agents still struggle to generate faithful formal specifications independently. Open article →