ThriftAttention: Selective Mixed Precision for Long-Context FP4 Attention

ThriftAttention is a two-stage mixed-precision attention method for long-context inference.

It selects a small set of critical query-key block pairs to compute in FP16, while the rest run in FP4.

Using FP16 for about 5% of blocks recovers much of the quality gap between FP4 and FP16, especially on long sequences.

The method merges the two paths with online softmax and adds little overhead across models and benchmarks.

This makes low-bit long-context models faster, cheaper, and closer in quality to full-precision attention.