Parallax: Parameterized Local Linear Attention for Language Modeling

Researchers introduced Parallax, a scalable local linear attention method for LLMs that removes the numerical solver and adds a learned projector.

The design is paired with hardware-aware kernel optimizations, with decode kernels that rival or beat FlashAttention 2/3.

At 0.6B and 1.7B model scales, Parallax delivered consistent perplexity and downstream gains over matched baselines.

The work suggests a better tradeoff among training efficiency, numerical stability, and model quality than prior efficient attention methods.