Stop Choosing Between BM25 and Vector Search: Implement Hybrid Search with RRF

Combine dense embeddings and sparse keyword search using Reciprocal Rank Fusion to eliminate retrieval failure modes in production RAG systems.

The Bottleneck in Production

Most production RAG pipelines start with pure vector search. It works reliably during initial demos, but fails quietly once users begin searching for exact product IDs, error codes, or domain-specific identifiers.

Vector search translates text into semantic coordinate spaces. Because dense models optimize for high-level concepts, they tend to blur fine-grained details. A search for ERR-502-BAD-GATEWAY might retrieve general network troubleshooting docs instead of the exact runbook for error 502.

Conversely, relying purely on keyword search (BM25) breaks when users paraphrase. A query for "reduce database memory footprint" completely misses an article titled "Mitigating PostgreSQL RAM Saturation" if there is no direct keyword overlap.

# The Naive Approach: Semantic-only retrieval misses exact tokens
def get_context(query: str, vector_db) -> list[str]:
    # Fails when query contains exact SKUs, UUIDs, or specific error logs
    return vector_db.similarity_search(query, k=5)

Relying on a single retrieval strategy creates hard blind spots that degrade downstream LLM generation quality.

The Architecture: Concurrent Hybrid Search & RRF

The fix is running sparse (BM25) and dense (vector) searches concurrently and merging their ranked outputs using Reciprocal Rank Fusion (RRF).

                     ┌───> [ BM25 Keyword Search ] ───> Sparse Ranked List ───┐
[ User Query ] ──────┤                                                         ├───> [ RRF Fusion ] ───> Top-K Documents ───> LLM
                     └───> [ Dense Vector Search ] ───> Dense Ranked List  ───┘

RRF avoids the core challenge of hybrid search: score normalization. BM25 produces unbounded positive floating-point scores, while vector search usually outputs cosine similarities between -1 and 1. Merging raw scores requires fragile heuristic scaling factors.

Instead of comparing raw scores, RRF scores documents based strictly on their relative rank position across both search lists:

$$RRF(d) = \sum_{m \in M} \frac{1}{k + r_m(d)}$$

Where:

• $M$ is the set of retrieval systems (BM25 and Vector).
• $r_m(d)$ is the 1-based rank position of document $d$ in system $m$.
• $k$ is a smoothing constant (typically set to 60) that prevents low-ranking outliers from disproportionately skewing results.

The Implementation

Here is a minimal, production-ready implementation combining BM25, dense embeddings, and RRF in under 25 lines of Python:

import numpy as np
from rank_bm25 import BM25Okapi
from sentence_transformers import SentenceTransformer
from sklearn.metrics.pairwise import cosine_similarity

def hybrid_rrf_search(query: str, corpus: list[str], top_n: int = 3, k: int = 60) -> list[str]:
    # 1. Sparse BM25 scoring & ranking
    bm25 = BM25Okapi([doc.lower().split() for doc in corpus])
    bm25_rank = np.argsort(bm25.get_scores(query.lower().split()))[::-1]

    # 2. Dense Vector scoring & ranking
    model = SentenceTransformer('all-MiniLM-L6-v2')
    doc_embs, query_emb = model.encode(corpus), model.encode([query])
    dense_rank = np.argsort(cosine_similarity(query_emb, doc_embs)[0])[::-1]

    # 3. Reciprocal Rank Fusion
    rrf_scores = {}
    for rank_idx, doc_idx in enumerate(bm25_rank):
        rrf_scores[doc_idx] = rrf_scores.get(doc_idx, 0.0) + (1.0 / (k + rank_idx + 1))
    for rank_idx, doc_idx in enumerate(dense_rank):
        rrf_scores[doc_idx] = rrf_scores.get(doc_idx, 0.0) + (1.0 / (k + rank_idx + 1))

    sorted_docs = sorted(rrf_scores.items(), key=lambda x: x[1], reverse=True)
    return [corpus[doc_idx] for doc_idx, _ in sorted_docs[:top_n]]

This pattern guarantees resilience. If a user inputs a technical keyword match, BM25 pushes the correct document to the top. If a user describes a high-level conceptual problem, dense