Where we left off

Lesson 19 put a real number on grading's cost: one model call per chunk (or per strip), scaling with k. One easy win is grading fewer chunks in the first place, without giving up any judgment quality on the ones that matter. This lesson adds a similarity-score pre-filter: a cheap threshold check, entirely local (no model call), that runs before the LLM grader even sees a chunk.

A pre-filter, not a replacement

This is not naive_rag Lesson 14's threshold repurposed to make final decisions, it's a much looser version of the same idea, used only to rule out chunks that are obviously, cheaply, unrelated by score alone. Anything that clears the bar still goes through the real LLM grader (Lessons 3, 12), the pre-filter's only job is skipping grading calls that would almost certainly come back "not relevant" anyway.

The code, piece by piece

PRE_FILTER_MIN_SCORE = 0.55

Chosen the same way naive_rag Lesson 14's threshold was: just above the observed "unrelated sentence" baseline, not from a formula. Kept loose on purpose, a pre-filter that's too aggressive risks dropping a chunk grading would have correctly kept, which defeats the entire point of adding grading in the first place.

with_prefilter = [c for c in all_scored if c["score"] >= PRE_FILTER_MIN_SCORE]
...
grades = {c["source"]: grade_chunk(question, c["text"]) for c in with_prefilter}

Only chunks that clear the pre-filter get an actual grading call. The ones that don't are treated as not-relevant without ever calling the model, saving exactly the calls Lesson 19 showed add up.

Checkpoint

  • A pre-filter trades a small amount of recall risk (a genuinely relevant chunk scoring just under the bar) for a real reduction in grading calls, kept loose specifically to make that risk small.
  • This is one of two direct responses to Lesson 19's cost problem, the other being Lesson 13's caching, both apply at once in a real system.
  • The LLM grader still makes every real relevance decision, the pre-filter only decides which chunks are cheap enough to skip entirely.

If anything here still feels unclear, ask before moving to Lesson 21.