β-OPSD: Deriving with Policy Optimization, Training with Self-Distillation

Researchers propose β-OPSD, a self-distillation method for reasoning-model training that generalizes on-policy self-distillation with a tunable KL weight.

The method derives a closed-form policy that mixes the reference and teacher policies, then realizes it with efficient logit interpolation and return-to-go credit assignment.

On math reasoning benchmarks, β-OPSD is more stable than vanilla OPSD and achieves better downstream accuracy.

The approach offers a more principled, practical way to combine policy optimization and self-distillation while reducing brittleness and improving reasoning performance.