OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond

Researchers introduced OScaR, a lightweight KV cache quantization framework for long-context LLMs.

It uses canalized rotation and omni-token scaling to address token norm imbalance and enable near-lossless low-bit compression.

Reported results include up to 3.0x faster decoding, 5.3x lower memory use, and 4.1x higher throughput versus BF16.

The method is designed for efficient deployment in long-context and multimodal models, with support from FlashDecoding-v2 and CUDA kernels.