Shreya Shankar explains limits of automated AI evals

youtube.com10 min read

Computer science professor Shreya Shankar explains why fully automating AI evaluations with AI is currently unachievable due to epistemic limitations and the subjective nature of product quality. The author argues that developers should instead view AI tooling as a way to express and apply their own human judgment faster across the evaluation life cycle of error analysis, measurement, and improvement. Open article →