Make Each Token Count: Towards Improving Long-Context Performance with KV Cache Eviction

Researchers proposed a learnable global KV cache eviction method for long-context inference.

It uses shared calibration so tokens from different layers and heads compete under one memory budget.

The approach cuts KV memory while matching or beating full-cache performance on language, vision-language, and dialogue tasks.

It tackles a key bottleneck in long-context AI and shows selective pruning can outperform keeping every token.