NVIDIA AI Releases Gated DeltaNet-2: A Linear Attention Layer That Decouples Erase and Write in the Delta Rule

NVIDIA introduced Gated DeltaNet-2, a 1.3B-parameter linear attention model trained on 100B FineWeb-Edu tokens.

It updates the delta rule to separate memory erasure and writing with distinct channel-wise gates for key and value channels.

The design keeps channel-wise decay from KDA while making state editing more expressive without relying on a larger KV cache.

NVIDIA says Gated DeltaNet-2 outperforms Mamba-2, Gated DeltaNet, KDA, and Mamba-3 on the referenced benchmark suite.