Advantage Collapse in Group Relative Policy Optimization: Diagnosis and Mitigation

Researchers found that GRPO can suffer from “advantage collapse,” where homogeneous reward batches produce ineffective gradients and stall learning.

They introduced Advantage Collapse Rate (ACR) to quantify how often training batches become ineffective.

To address it, they proposed AVSPO, which adds virtual reward samples to reduce collapse without extra rollouts.

AVSPO improves training stability and benchmark performance for large language models across model sizes.