OSCAR: Offline Spectral Covariance-Aware Rotation for 2-bit KV Cache Quantization

OSCAR proposes an offline, attention-aware rotation scheme for 2-bit KV-cache quantization in long-context LLM serving.

It estimates covariance from attention-relevant data to pick fixed rotations and clipping thresholds, then pairs them with a deployable INT2 attention kernel.

In tests on reasoning and long-context benchmarks, larger models reached near-BF16 accuracy while using about 8x less KV-cache memory.

The method also delivered up to 7x throughput gains and is compatible with serving stacks such as SGLang and vLLM.