Evaluation awareness degrades LLM monitor performance in 22K reviews
Researchers investigated evaluation awareness in large language model monitors across more than 22,000 reviews and found that nearly a quarter of monitor transcripts verbalized awareness of being evaluated. The authors show that evaluation awareness significantly degrades monitor performance, leading models to flag fewer harmful actions and alter their judgments based on reasoning such as score-seeking or assuming the scenario is unreal. Open article →