Metadata-Aware Hybrid Retrieval

Where we left off

Every lesson so far searched this course's entire six-document corpus. A real corpus has thousands of documents, and a user often already knows the scope they want, "just my network notes," "just this project." This lesson adds that filter, applied identically to both retrievers before they score anything, not after.

The code, piece by piece

CATEGORY = {
    "home_network.md": "network",
    "old_travel_router.md": "network",
    ...
}

One tag per document, the same idea as naive_rag Lesson 12's source filter, a hand-assigned category instead of the filename itself.

def hybrid_search(query, all_names, token_lists, doc_vectors, category=None):
    candidates = [n for n in all_names if category is None or CATEGORY[n] == category]
    dense = dense_ranking(query_vector, candidates, [doc_vectors[n] for n in candidates])
    sparse = sparse_ranking(query, candidates, [token_lists[n] for n in candidates])
    return reciprocal_rank_fusion([dense, sparse])

The filter happens once, before either retriever runs, both dense and sparse only ever see the scoped-down candidate list. That's the whole point of doing it here instead of filtering the fused result afterward: an out-of-scope document can never sneak into the top-k through one retriever just because it happened to fuse well, it's excluded from scoring entirely.