Google's TurboQuant reduces AI LLM cache memory capacity requirements by at least six times — up to 8x performance boost on Nvidia H100 GPUs, compresses KV caches to 3 bits with no accuracy loss

TurboQuant was published by Google Research, google Research published TurboQuant, a training-free compression algorithm that quantizes LLM KV caches to 3 bits without model accuracy loss.

Delivered up to 8x performance increase on Nvidia H100 GPUs 4-bit TurboQuant provided up to eight-times faster attention logits computation compared to unquantized 32-bit keys on Nvidia H100 GPUs.

Reduced KV cache memory requirements by at least 6x, turboQuant reduced KV cache memory by at least six times in benchmarks.

Quantizes KV caches to 3 bits, turboQuant compresses KV caches down to 3 bits without any loss in model accuracy.

PolarQuant transforms data vectors into polar coordinates to skip per-block normalization, polarQuant converts vectors from Cartesian to polar coordinates and separates each vector into a radius and angles to avoid per-block normalization overhead.