Context Memorization for Efficient Long Context Generation

Researchers introduced attention-state memory, a training-free method for efficiently storing long prompt context in large language models.

It precomputes and keeps attention states for prefix-query pairs in lightweight external memory, preserving prefix influence during generation.

The approach reduces inference latency and memory footprint versus full-attention and other long-context methods.

It also improves accuracy under limited memory, making long-context generation cheaper and faster for LLMs such as LLaMA-3.1-8B.