Hybrid RAG: Combining Dense (Semantic) and Sparse (Keyword) Search

Two kinds of search, run together

Hybrid RAG runs two retrieval methods in parallel and merges their results: dense retrieval (embedding-based vector search, covered in Lesson 3) for semantic similarity, and sparse retrieval, most commonly the decades-old algorithm BM25, for exact keyword and term-frequency matching. Sparse retrieval scores a chunk by how often and how distinctively a query's literal terms appear in it, the same basic idea a traditional search engine uses, with no notion of meaning at all.

Why combining them beats using either alone

Dense and sparse retrieval fail in complementary ways: dense search misses exact terms it has never seen phrased that way, sparse search misses paraphrases and synonyms it has no notion of meaning for. A hybrid system computes both a dense score and a sparse score for every candidate chunk, then merges the two rankings, commonly with an algorithm called Reciprocal Rank Fusion (RRF), which combines two ranked lists based on each item's rank position rather than trying to make two differently-scaled scores directly comparable.

The practical effect: a query containing both a technical term and a natural-language description ("error E4021 keeps happening after login") benefits from sparse search catching "E4021" exactly and dense search catching the semantic connection between "keeps happening" and words like "recurring" or "intermittent" that the actual document might use instead.

Cost of the improvement

Hybrid retrieval roughly doubles the retrieval-side computation, running two searches and a fusion step instead of one, and requires maintaining two indexes over the same data instead of one. For a knowledge base heavy on codes, names, or precise technical vocabulary, most teams find that cost well worth it; for prose-heavy, conversational content, the gain over dense-only search is often smaller.