Tilde Research Introduces Aurora: A Leverage-Aware Optimizer That Fixes a Hidden Neuron Death Problem in Muon

Tilde Research introduced Aurora, a new optimizer that addresses a flaw in Muon that can leave many neurons permanently inactive in tall MLP weight matrices.

Aurora builds on ideas from NorMuon and U-NorMuon to preserve orthogonal updates while using leverage-aware normalization to reduce neuron death.

The team reports a 1.1B-parameter pretraining experiment and a new state-of-the-art result on the modded-nanoGPT speedrun benchmark.

Code for Aurora has been released, with the goal of improving large-model training stability and efficiency.