Nous Research Proposes Lighthouse Attention: A Training-Only Selection-Based Hierarchical Attention That Delivers 1.4–1.7× Pretraining Speedup at Long Context

Nous Research introduced Lighthouse Attention, a training-only hierarchical sparse attention method for long-context models.

It pools queries, keys, and values into a multi-level pyramid, then selects a subset outside the attention kernel before running standard FlashAttention.

Against a cuDNN-backed SDPA baseline, it delivered 1.40× to 1.69× faster pretraining.

Final training loss was similar or lower, suggesting the speedup does not sacrifice model quality.