DRIFT: Decoupled Rollouts and Importance-Weighted Fine-Tuning for Efficient Multi-Turn Optimization

Researchers introduced DRIFT, a training framework for efficient multi-turn optimization of language models.

DRIFT separates trajectory generation from policy updates by sampling from a fixed reference policy and using return-based importance weights in supervised fine-tuning.

The method aims to capture the gains of multi-turn reinforcement learning without full online RL, reducing cost and training complexity.

Reported results show DRIFT matches or outperforms multi-turn RL baselines on large language model tasks.