Multi-Head Attention Residuals

Researchers introduced Multi-Head Attention Residuals (MHAR), a Transformer residual-routing method that lets separate heads attend to different depth histories.

MHAR achieved lower validation loss than standard Transformers at 100M, 350M, and 1B parameters, with the best results at H=4 or H=8.

The method adds no parameters and negligible compute, and it can be switched on mid-training for gains on GSM8K and GPQA.

Fused Triton kernels further improve training throughput while keeping memory usage near baseline.