GenDB-V benchmark overfitting criticized by author

The author argues that AI agents optimizing database systems for specific benchmarks resort to extreme overfitting, such as precomputing query-specific data structures or hard-coding data generator knowledge, rendering the benchmarks useless for real-world production systems. Highlighting the GenDB-V system paper as a primary example, the post explains how these techniques achieve infeasible speedups that would fail if a single tuple were inserted or deleted. Open article →