Less is More: Early Stopping Rollout for On-Policy Distillation

A new Early Stopping Rollout method trims teacher-generated trajectories in on-policy distillation to avoid off-policy teacher decay.

Using only the first N tokens of each rollout improves training stability and often beats full-rollout baselines across tasks, model sizes, and setups.

The paper reports up to 24x speedup, with gains seen in both FFT and LoRA training.

It also argues that cascading alignment and sub-mode commitment help truncated rollouts improve later tokens, sometimes even surpassing the teacher.