Google Introduces TurboQuant: A New Compression Algorithm that Reduces LLM Key-Value Cache Memory by 6x and Delivers Up to 8x Speedup, All with Zero Accuracy Loss

TurboQuant was proposed by Google Research, google Research proposed TurboQuant, a data-oblivious quantization framework for KV cache compression.

KV cache can be reduced by 6x, turboQuant reduces KV cache memory by about 6x and can deliver up to 8x speedups according to the announcement.

TurboQuant applies a random rotation and per-coordinate scalar quantization, turboQuant applies a random rotation to input vectors and then performs per-coordinate scalar quantization.

Has MSE distortion within ≈2.7× of the theoretical lower bound, turboQuant’s MSE distortion is within a small constant factor (≈2.7) of the information-theoretic lower bound across bit-widths.

Has MSE distortion 0.36 at b=1, at bit-width b=1, TurboQuant’s MSE distortion is about 0.36 compared with the lower bound 0.25.