We need more transparency about alignment overfitting

The author argues that a lack of public transparency from AGI companies regarding their alignment methods makes it difficult to assess long-term risks, as available evidence is consistent with companies overfitting on alignment data. While some companies like Anthropic, OpenAI, and Google DeepMind report taking precautions to separate evaluation and training distributions, others have faced incidents involving accidental training on evaluation data or monitoring transcripts. The author contends that publishing alignment approaches, conducting out-of-distribution generalization research, and involving third-party evaluators are necessary to properly evaluate misalignment risks. Open article →