Where Hand-Rolled Retrieval Breaks Down

Where we left off

naive_rag Lesson 19 timed a hand-rolled dense linear scan as its corpus grew. This course has two hand-rolled retrievers to worry about, and they don't scale the same way. This lesson times both, side by side, with a synthetic corpus (random vectors, random tokens, same idea as naive_rag Lesson 19, real embedding calls for thousands of documents would be slow and pointless here).

The code, piece by piece

def idf(term, documents) -> float:
    doc_count = sum(1 for doc in documents if term in doc)
    return math.log(...)

Every call to idf() rescans the entire document collection to count how many documents contain this one term, exactly as it's been written since Lesson 4. That was fine at six documents. It stops being fine once sparse_hand_rolled() calls it once per query term, per document being scored, on every single search.

for doc in documents:
    for term in query_tokens:
        ...
        score += idf(term, documents) * (...)

Two nested loops, and the inner one calls a function that's itself another full scan of documents. That's the shape that gets expensive fast, not because BM25 the algorithm is slow, but because this particular hand-rolled implementation never caches anything.