Bug or Feature^2: Weight Drift, Activation Sparsity, and Spikes

A formal and empirical study found that gradient descent with asymmetric activations tends to push weights negative across multiple model types.

That negative weight drift increases the number of zeroed ReLU activations, linking training dynamics to sparsity in modern networks.

In GPT-nano experiments, ReLU² and GELU² improved results, but ReLU² also caused large activation spikes.

The spike problem can be mitigated with clipping, suggesting a simple way to gain performance while controlling instability.