Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

Researchers proposed symmetry-compatible optimizers that make parameter updates equivariant to each layer’s symmetry group.

The method was extended beyond matrices to embeddings, LM heads, SwiGLU projections, and MoE routers.

In pretraining runs, the approach achieved lower validation loss than AdamW and sometimes improved training stability.

Results were reported on models including Qwen3-0.6B, Gemma 3 1B, OLMoE-1B-7B, and gpt-oss, suggesting broader use for dense and sparse LMs.