Demystifying Data Organization for Enhanced LLM Training

Researchers found that the order of training data can significantly affect LLM training stability and performance.

They identified four useful ordering principles: boundary sharpening, cyclic scheduling, curriculum continuity, and local diversity.

Using precomputed sample scores, they proposed two methods, STR and SAW, to organize data more strategically.

Experiments across model sizes and data scales showed gains in both pre-training and supervised fine-tuning.