KV Caching in LLMs

KV caching speeds up LLM decoding by reusing past key and value tensors instead of recomputing them at every step.

The first token is still slower because the prompt must go through a prefill phase before its states can be cached.

This improves response latency, but the tradeoff is significantly higher GPU memory use per request.

That memory cost affects scalability and serving economics for models like ChatGPT, Claude, and Qwen 2.5 72B.

Related optimizations such as grouped-query attention, multi-query attention, and paged attention help manage these costs.