RAG retrieval drops on 2.68M Wikipedia distractors

The author tested how retrieval-augmented generation search performance changes as collection sizes and distracting passages grow, finding that BM25 and embedding setups scaled well on a 2,000-paper set but experienced severe performance drops across 2.68 million Wikipedia passages. The tests showed that hard search methods hurt downstream answer models primarily by pushing correct passages out of the top results. Open article →