CONF-KV: Confidence-Aware KV Cache Eviction with Mixed-Precision Storage for Long-Horizon LLM

Researchers introduced CONF-KV, a confidence-aware KV cache manager for long-context LLM inference.

It uses next-token confidence to adjust cache budgets, then evicts tokens using attention and recency signals while protecting a recent window.

The system also combines mixed-precision storage and attention optimizations to further reduce memory use.

Across multiple models and long sequences, it used memory similar to a 512-token sliding window and stayed within about 1.5 to 2.1 perplexity points of full KV.

It outperformed sliding-window and H2O baselines on retrieval and task benchmarks, including Needle-in-a-Haystack and VisualWebArena.