Agents are users, so your software needs evals

The author explains how to build end-to-end evaluations (evals) for software used by AI agents, introducing a testing pyramid and a step-by-step framework using the open-source tool Harbor. The article details how to set goals, define success and efficiency metrics like Pass@k and Pass^k, design task sets, run trials using Vercel infrastructure, and analyze failure traces to improve agent reliability. Open article →