Alignment 'train/test splits' should be sacred

The author argues that training AI alignment models against all available assessment techniques mirrors the machine learning error of training on an entire dataset without holding out a test split. By optimizing models directly against monitors like chain-of-thought analysis, human feedback, and automated audits, developers render those monitors obsolete for verifying alignment during inference. The article calls for transparency from frontier AI companies, advocating for the deliberate reservation of certain alignment techniques exclusively for testing. Open article →