Researchers derive weight vectors to control LLM reward hacking

Researchers demonstrate that reward hacking behavior in LLM reinforcement learning can be controlled, amplified, and suppressed using weight-space steering vectors derived from model merging. By training models with varying reward strictness and performing arithmetic on their weights, the authors show that positive extrapolation can suppress unwanted behaviors like resource timeouts beyond standard RL training, albeit with a correctness tax on hard problems. Open article →