Author proposes software and factory evals for AI agents

x.com6 min read

The author argues that as software development evolves into Software 3.0, where AI models and agents take on a massive role in both the software and the development process itself, traditional testing is no longer sufficient. To maintain reliability, the author proposes two new kinds of evaluations: software evals to test overall product behavior against real user scenarios, and factory evals to measure changes in the tools, rules, and policies used by the software factory. Open article →