Grading All Top-k Chunks

Where we left off

Lesson 3 graded exactly one chunk, the top-1 result, and confirmed it was wrong. But grading only the top result means the correct chunk (bookshelf.md, ranked second by score) never even gets a chance, grading can't rescue a chunk it never sees. This lesson over-fetches before grading: retrieve k=3 instead of k=1, then grade every one of them.

Why over-fetch first

This is the move that makes correction-by-filtering possible at all. If retrieval only ever returns its single best guess, grading that one chunk "not relevant" leaves you with nothing, no better off than before. Fetching a few extra candidates up front, then grading all of them, means a correct-but-not-top-ranked chunk (exactly bookshelf.md's situation here) is still in the pool when Lesson 5 filters by grade.

The code, piece by piece

K = 3
top_k = retrieve(QUESTION, store, K)

Same retrieve() as Lesson 2/3, just called with a larger k. Nothing about retrieval itself changed, correction happens after this point, not by making retrieval smarter.

def grade_all(question: str, chunks: list[dict]) -> list[dict]:
    return [{**chunk, "grade": grade_chunk(question, chunk["text"])} for chunk in chunks]

Lesson 3's grade_chunk(), called once per retrieved chunk instead of once. Each chunk keeps its score and gains a grade, both pieces of information survive into Lesson 5.