Mergeable Model-Side Aggregation States for Long-Context Language Models

Researchers propose a frozen-model aggregation interface that extracts canonical identities and stores them in mergeable HyperLogLog states.

The state stays fixed at 2 KiB, can be merged across up to 256 context segments, and preserves identical readouts after merging.

On a one-million-record distinct-count test, the method achieved low error and outperformed direct reasoning and chain-of-thought baselines on aggregate-then-reason tasks.

The approach offers a practical way to do set-based aggregation inside long-context language models without extra generate-execute steps.