Directional Alignment Mitigates Reward Hacking in Reinforcement Learning for Language Models

Researchers framed reward hacking in language-model reinforcement learning as a geometry problem in parameter updates.

They found that shortcut exploitation is linked to larger directional shifts in model updates.

Their trusted-direction projection method keeps gradients near a clean reference subspace.

The approach can delay reward hacking while preserving task performance, improving training reliability.