Precise AI Scientific Workfows For All Tasks

Compare BM25 and LLM-based vector embeddings for information retrieval

Abstract

This review contrasts BM25, a sparse lexical ranking function rooted in probabilistic IR, with LLM-based (dense) vector embeddings used for semantic retrieval. We summarize modeling differences, empirical trends across standard benchmarks, efficiency/engineering trade-offs, domain/multilingual considerations, and open problems. Evidence across MS MARCO, TREC Deep Learning, and BEIR suggests hybrids—sparse + dense—often yield the best effectiveness–efficiency balance.

1. Background

BM25. A term-matching method from the probabilistic relevance framework; scores documents by TF-IDF-like signals with length normalization (Robertson & Zaragoza, 2009). Advantages include simplicity, interpretability, robustness, and low cost.
Dense/LLM embeddings. Neural encoders (Bi-encoders like DPR; late-interaction like ColBERT; or general LLM/embedding models) map text to high-dimensional vectors; retrieval uses vector similarity via ANN indexes. They capture paraphrase and semantic similarity beyond exact term overlap (Devlin et al., 2019; Karpukhin et al., 2020; Khattab & Zaharia, 2020).

2. Modeling Differences

• Signal type: BM25 relies on exact token overlap; dense models use distributed semantics.

• Training: BM25 is training-free; dense retrieval typically requires supervised (MS MARCO) or distillation/contrastive pretraining.

• Ranking pipeline:

  • Sparse first-stage (BM25) → Neural re-ranker (cross-encoder) is a common strong baseline (Nogueira & Cho, 2019).

  • Dense first-stage can replace or complement BM25; late-interaction (ColBERT) preserves some token granularity for accuracy at higher cost.

3. Empirical Findings (high level)

• On keyworded or head queries, BM25 remains highly competitive; exact matches matter.

• On conversational/semantic queries and mismatch vocab (synonyms, paraphrases), dense retrieval typically outperforms BM25.

• Zero-shot/transfer (BEIR): dense retrievers can generalize, but performance varies by domain; hybrids reduce variance (Thakur et al., 2021).

• Reranking: Cross-encoders (e.g., monoBERT) over BM25 candidates often surpass pure dense retrieval in effectiveness, at higher latency.

4. Efficiency & Engineering

• Indexing & memory:

  • BM25: inverted indexes are compact; scales easily on CPU.

  • Dense: vector stores (FAISS, HNSW) require larger memory/compute.

• Latency:

  • BM25 is milliseconds-fast.

  • Dense first-stage is fast with ANN, but building indexes and updating them is heavier; late-interaction models (ColBERT) cost more at query time.

• Interpretability: BM25 scores are explainable (term contributions). Dense scores are opaque; attribution requires auxiliary tooling.

5. Domain, Multilingual, and Robustness

• Domain shift: BM25 degrades gracefully; dense models may require domain-adaptive finetuning or unsupervised adaptation.

• Multilingual: Multilingual embeddings enable cross-lingual retrieval (query ↔ doc in different languages) with no translation step; BM25 typically needs per-language indexes or MT preprocessing.

• Robustness: BM25 is less sensitive to adversarial paraphrase but brittle to vocabulary mismatch; dense is the reverse.

6. Evaluation Practices

Common datasets/benchmarks: MS MARCO (passage/document), TREC Deep Learning, BEIR (zero-shot transfer across 18+ tasks). Metrics: MRR@10, nDCG@10, Recall@k, MAP. For production, report both effectiveness and cost (latency, memory, $$ per 1k queries).

7. When to Use What

• Prefer BM25 when: queries are short/keyworded; infrastructure must be lightweight; explainability matters; frequent index updates are needed.

• Prefer Dense when: queries are natural-language; semantic recall matters (QA, support search, research); cross-lingual retrieval is required.

• Prefer Hybrid when: you need strong out-of-the-box performance across mixed query types and domains—BM25 (or SPLADE) for candidate generation + dense rerank (bi-encoder or cross-encoder).

8. Open Problems & Trends

• Cost-effective hybrids: dynamic routing (choose sparse vs dense per query).

• Lightweight rerankers: distilled cross-encoders for near-cross-encoder quality at lower latency.

• Continual/domain adaptation: self-supervised and synthetic-labeling pipelines to keep embeddings fresh.

• Safety & bias: auditing dense retrievers for demographic or topical skew; robust evaluation beyond MS MARCO.

• Structured + unstructured fusion: retrieval over tables/graphs + text with unified embeddings.

9. Practical Recipe (production-friendly)

1. BM25 (or SPLADE) top-1k → 2) bi-encoder dense rerank to top-100 → 3) cross-encoder rerank to top-20 for UI.
   Add query rewriting (for abbreviations/typos), caching, and telemetry (query difficulty, fallback to BM25 on ANN miss). For multilingual, use multilingual encoders and per-language BM25 as a backstop.

10. Short Annotated Reading List (starting points)

• Robertson & Zaragoza (2009): BM25 and probabilistic IR.

• Devlin et al. (2019): BERT—foundation for modern neural rerankers.

• Nogueira & Cho (2019): MS MARCO passage re-ranking with BERT (strong two-stage baseline).

• Karpukhin et al. (2020): DPR dense passage retrieval (bi-encoder).

• Khattab & Zaharia (2020): ColBERT late interaction (accuracy/efficiency trade-off).

• Thakur et al. (2021): BEIR—zero-shot IR benchmark across diverse tasks.

If you want, I can tailor this into a 1-page PDF with a comparison table (pros/cons, datasets, metrics) or adapt it to your specific domain (e.g., research papers, customer support, code search).