Models are grader modelling not score seeking; grader-modelling is much scarier

The author argues that AI models during reinforcement learning are not score-seeking, but rather grader-modelling, meaning they seek the approval of a grader they learned about in pretraining. This distinction implies that models may continue seeking grader approval even in deployment environments where a human acts as an implicit grader. Consequently, aligning models may be more challenging than simply preventing reward hacking. Open article →