Most Retrieval-Augmented Generation (RAG) systems suffer from a fundamental “retrieval gap.” They either understand semantic meaning but miss exact keyword matches, or they find specific terms but lose the broader context of the query.
To build production-grade AI, you cannot rely on a single retrieval method. You need a multi-stage pipeline that combines the strengths of different search technologies while using mathematical fusion to resolve their differences.
The Real Difference: Dense Vector Search vs. BM25 Keyword Search
To understand why hybrid search is necessary, we must first look at the two pillars of information retrieval:
- Dense Vector Search: This method uses embeddings to capture semantic meaning. It understands that “refund” and “money back” are related, even if they share no common characters. However, it often struggles with exact matches like product serial numbers or specific technical terms.
- BM25 (Sparse) Search: This is a keyword-based approach. It is incredibly effective at finding exact matches for specific strings but lacks any understanding of semantic context or synonyms.
By combining these two, you ensure your system is both semantically aware and keyword-accurate.
Why Raw Scores Can’t Be Compared: Reciprocal Rank Fusion (RRF)
A common mistake in hybrid search is attempting to add raw scores from different systems together. You cannot mathematically combine a cosine similarity score (typically between 0 and 1) with a BM25 score (which can be any positive number). They are on entirely different scales.
Reciprocal Rank Fusion (RRF) solves this by ignoring the raw scores entirely and focusing solely on the rank position of an item in each list. The formula for calculating the fused score is:
Score = (1 / (k + rank_dense)) + (1 / (k + rank_sparse))
Where k is a constant (usually 60) that prevents low-ranked items from having an outsized influence. This allows us to merge results from Qdrant and BM25 into a single, unified list without needing to normalize their scores.
Retriever vs. Reranker: The Two-Stage Approach
It is vital to understand that a reranker is fundamentally different from a retriever. A retriever’s job is to cast a wide net and find the top 50 or 100 most likely candidates. It is built for speed and scale.
A reranker, however, acts as a precision tool. Instead of looking at thousands of documents, it takes the small shortlist provided by the retriever and performs a deep, computationally expensive analysis on each candidate. While a retriever uses fast vector math, an LLM-based reranker acts as a cross-encoder, reading both the query and the document chunk together to assign a highly accurate relevance score.
Evaluating Your RAG Pipeline
You cannot improve what you do not measure. To move beyond trial-and-error, you must evaluate your retrieval strategy using standardized metrics. If you are looking for more ways to validate your AI outputs, check out our guide on Moving Beyond Trial-and-Error: Mastering LLM Evaluation with Promptfoo.
| Metric | What it tells you |
|---|---|
| Hit Rate | Did the retriever find at least one relevant chunk in the top K results? |
| MRR (Mean Reciprocal Rank) | How high up in the list is the first relevant chunk located? |
| NDCG | Are the most relevant chunks placed at the very top of the ranking? |
The Production-Style RAG Architecture
A robust, production-ready codebase should follow a structured flow to ensure scalability and testability. The architecture implemented in the full implementation of this hybrid search framework follows these stages:
- Ingestion: Extract text via
pdfplumber, generate embeddings for Qdrant, and build a BM25 index. - Sparse Retrieval: Use
rank_bm25to find keyword matches. - Dense Retrieval: Query the vector database for semantic matches.
- Fusion (RRF): Merge results using rank-based scoring.
- Reranking: Use an LLM to reorder the top candidates based on deep relevance.
- Generation: Pass the refined context to an LLM for final answer generation.
Getting Started with ReRankEval
You can replicate this entire pipeline using Python. First, set up your environment and install the dependencies:
git clone https://github.com/saurabhkamal/Hybrid-Search-RRF-Reranking-System
cd ReRankEval
python -m venv .venv
source .venv/bin/activate # or .venv\Scripts\activate on Windows
pip install -r requirements.txtOnce your environment is ready, follow these three steps to test the system:
- Ingest: Build your indexes with
python -m rag.ingest - Evaluate: Run a comparison of different strategies with
python -m eval.run_comparison - Generate: Ask questions end-to-end using
python -m rag.generate
By implementing this multi-stage approach, you move from simple keyword matching to a sophisticated system capable of understanding complex user intent with high precision.
Ready to optimize your RAG pipeline? Clone the repository today and start benchmarking your retrieval strategies!
