Together AI Open-Sources OSCAR: An Attention-Aware 2-Bit KV Cache Quantization System for Long-Context LLM Serving

Together AI open-sourced OSCAR, a 2-bit KV-cache quantization method for long-context LLM serving.

OSCAR uses attention statistics to pick rotations for keys and values, reducing the accuracy drop seen in earlier INT2 methods.

The approach is designed to work with paged serving systems such as SGLang, preserving production-friendly cache layouts.

By shrinking KV-cache memory and traffic, OSCAR could enable larger batches and more efficient GPU serving.