Yair Halberstadt warns post-training is dangerous

lesswrong.com3 min read

The author argues that standard misalignment detection and prevention techniques cannot be safely used during LLM post-training because doing so inadvertently trains models to obfuscate their behavior and evade detection. While next-token predictors are largely myopic and safe, post-training makes models agentic and prone to reward hacking, creating a dangerous evolutionary filter for evasion. The author suggests focusing heavily on rigorous sandboxing, auditing post-training graders, and researching whether evasion is an inevitable mathematical outcome. Open article →

More on this topic