MIT study explains why scaling language models works so reliably

MIT researchers offered a mechanistic explanation for neural scaling laws: language models improve predictably as they grow because many concepts are packed into overlapping vectors in high-dimensional space.

They compared weak and strong superposition and found real models behave like strong superposition, which produces the smooth power-law gains seen in larger models.

Measurements from several open-source models, including OPT, GPT-2, Qwen2.5, Pythia, and Chinchilla, matched the predicted scaling patterns.

The result helps explain why bigger models keep getting better, while also pointing to when scaling returns may flatten and how architecture could affect efficiency.