What we compared

Four setups, same queries, same corpus, same metrics:

• BM25. Sparse keyword ranking. Fast, training free, strong on exact terms like names, codes, and abbreviations.
• Dense. Semantic vector search using sentence embeddings. Strong on paraphrase and concept matches, weaker on exact tokens.
• Hybrid. BM25 and dense fused with Reciprocal Rank Fusion, which combines by rank position rather than raw score, so the two scales do not need to be normalised.
• Hybrid + Rerank. A cross encoder rescoring the top 100 fused candidates. Slower per query, sometimes more accurate.

How it was measured

The setup is deliberately plain so anyone can reproduce it.

• Dataset: BEIR scifact, 5,183 documents, 300 test queries with relevance judgments (roughly one relevant document per query, which is why recall climbs fast with k).
• Embedding model: sentence-transformers/all-MiniLM-L6-v2.
• Reranker: cross-encoder/ms-marco-MiniLM-L-6-v2.
• Fusion: Reciprocal Rank Fusion, k = 60.
• Rerank depth: top 100 candidates.
• Metrics: Precision@k, Recall@k, nDCG@k for k in 1, 5, 10, plus MRR@10, plus mean query latency in milliseconds.
• Hardware and date: CPU only, no GPU, run July 2026. This matters a lot here, because the reranker latency below is dominated by the cross encoder running on CPU and would drop sharply on a GPU.

Latency is measured at query time (encode the query and search). One-time index building is excluded, because you pay it once, not per query.

Results

Method	nDCG@1	nDCG@10	P@1	Recall@10	MRR@10	Latency (ms)
BM25	0.523	0.652	0.523	0.776	0.618	15
Dense	0.503	0.645	0.503	0.783	0.605	27
Hybrid (RRF)	0.557	0.684	0.557	0.806	0.650	34
Hybrid + Rerank	0.577	0.691	0.577	0.821	0.659	1934

What the numbers mean

Three questions, answered by the run.

• Did hybrid beat both parents? Yes, clearly.
  Hybrid fusion scored 0.684 on nDCG@10, above BM25 at 0.652 and dense at 0.645, and it also led on recall@10, MRR, and top rank precision. Combining keyword and semantic retrieval beat either one alone, for a modest latency cost of 34ms against 15ms for BM25 and 27ms for dense. If you take one thing from this study, it is that fusion is close to free and reliably helps.

• How much did the reranker add, and at what cost? Almost nothing, very expensively.
  On nDCG@10 the reranker moved the score from 0.684 to 0.691, about one percent. Its help concentrated at the very top rank, where P@1 rose from 0.557 to 0.577, so it is worth considering if your product shows a single answer. But latency jumped from 34ms to 1934ms per query, roughly 56 times slower on this run. That penalty is hardware bound. A cross encoder is far faster on a GPU and with a smaller rerank depth, so treat the latency figure as a ceiling, not a constant. Decide the reranker on your latency budget, not on the leaderboard.

• Where did dense help or hurt?
  Dense on its own was the weakest value here. It trailed BM25 slightly on nDCG@10 and MRR while running about twice as slow, so as a standalone retriever the embedding model did not earn its place. It still pulled its weight inside the hybrid, since fusion beat BM25 alone, but the lesson is that a vector index is not automatically an upgrade over keyword search on every corpus.

When this doesn't hold

The result is not universal, and saying so is what makes it trustworthy.

• Small or uniform corpus. On a few hundred near identical documents, the gap between methods shrinks and the reranker earns its latency less.
• Pure exact match domains. For code search, part numbers, or legal citations, BM25 alone can match or beat the fancier setups.
• Tight latency budgets. The cross encoder cost 56 times the latency of plain hybrid in this run. Under a strict latency ceiling, and especially on CPU, hybrid without rerank is often the sensible stop.
• Different embedding model or reranker. Swap either and the numbers move. These results describe the specific models above, not dense retrieval in the abstract.
• Domain shift. An embedding model trained on general text can underperform on specialised corpora until it is adapted, which can flip the dense result.

Reproduce it

Everything here comes from one script, retrieval_benchmark.py.

Quick offline check that the pipeline runs:

python3 retrieval_benchmark.py --selftest

Real run:

python3 retrieval_benchmark.py \
    --corpus data/corpus.jsonl \
    --queries data/queries.jsonl \
    --qrels data/qrels.tsv \
    --rerank-depth 100

It writes results.csv and results_ndcg.png. Full data format and setup notes are in the README.

FAQ

What is Reciprocal Rank Fusion?
A way to merge two ranked lists using each document's rank position rather than its raw score. Because it ignores score scale, it fuses BM25 and dense results without any normalisation step, which makes it a robust default for hybrid search.

Why add a reranker if hybrid is already good?
A cross encoder reads the query and each candidate together, so it can judge relevance more precisely than a first stage retriever that scored them separately. In this run that precision showed up mostly at the very top rank, not in nDCG@10, and it came at a large latency cost. So a reranker is worth it when top rank accuracy matters and you have the latency budget, not as an automatic add on.

Does dense retrieval need a GPU?
Not for a corpus this size. Encoding runs on CPU, just slower. Index building is a one time cost. For large corpora a GPU and an approximate nearest neighbour index become worthwhile.

Which dataset should I use?
If you want a public, citable result, use a BEIR dataset like scifact or nfcorpus. If you want a result that sells your own work, run it on an anonymised slice of a real client corpus, which is far more persuasive to a buyer because it is their kind of data.

How do I pick k for the cutoff metrics?
Match k to how many results your product actually shows. If the UI surfaces five sources, P@5 and nDCG@5 matter more than @10.