Towards Robust Reinforcement Learning for Small-Scale Language Model Agents

Researchers evaluated 15 small language model and corpus pairings with PPO to study why reinforcement learning is unstable.

They found three recurring failure modes: policy collapse, reward hacking, and numerical issues such as bfloat16 overflow.

The paper proposes practical fixes including adapter reinitialization, float32 updates, and reward safety checks.

These changes improved training stability and increased win rates, making SLM alignment more reproducible.