Safety Alignment of LMs via Non-cooperative Games

bamos.github.io1 min read

Researchers introduce AdvGame, a new safety alignment framework that frames language model safety as a non-zero-sum game between jointly trained Attacker and Defender language models. Utilizing online reinforcement learning and a preference-based reward signal from pairwise comparisons, the method shifts the Pareto frontier to produce a Defender model that is both more helpful and more resilient. Additionally, the resulting Attacker model serves as a strong, general-purpose red-teaming agent for probing target models. Open article →