ChunkFT: Byte-Streamed Optimization for Memory-Efficient Full Fine-Tuning

Researchers introduced ChunkFT, a memory-efficient framework for full fine-tuning of large language models.

It streams parameters in chunks, activating only a working subset at a time and computing gradients for arbitrary sub-tensors without changing the model architecture.

The method uses far less GPU memory while maintaining strong runtime and optimization performance.

On tasks including language understanding, math reasoning, and MT-Bench, ChunkFT matches or beats existing memory-efficient baselines.

This could make high-quality full-model adaptation much cheaper and more accessible on limited hardware.