BM25 Wins at Scale: A Scaling Study of Retrieval-Augmented Generation Paradigms

A scaling study compared several retrieval-augmented generation methods across 28 nested corpus sizes under a fixed evaluation setup.

BM25 overtook the File-System Agent at around 10 million corpus tokens and stayed ahead at all larger scales.

Dense retrieval was cheaper but less accurate, while graph-based RAG struggled to scale as corpus size grew.

The results show that retrieval performance depends on corpus size, so single-scale benchmarks can be misleading.

Lexical retrieval appears to be the strongest scalable default, with agentic reasoning better used after ranked retrieval rather than instead of it.