Stochastic Gradient Descent (SGD’s) Frequency Bias and How Adam Fixes It

A controlled NumPy experiment trained a linear model on tokens spanning four orders of magnitude in frequency.

The results showed SGD favors frequent tokens, giving them far more learning progress than rare ones.

Adam, by using per-parameter gradient history, effectively increases step sizes for infrequently updated weights.

The finding explains why adaptive optimizers can help language models learn rare but important tokens faster.