Minimal Evaluation, Routed vs. Fixed

Where we left off

Lesson 16 showed misrouting happening, by hand, on two questions chosen because they'd misroute. That's convincing for a demo but it isn't a measurement: it doesn't say how often routing actually helps across a whole mix of questions, or what it costs to find out. This lesson replaces "here's a case where routing helped" with a small, repeatable comparison: run the same nine labeled questions from Lesson 7 (extended slightly here) through every fixed strategy alone, then through the router, and compare both precision@k and how many API calls each condition made.

What we're measuring

Four conditions, side by side, over the same nine questions:

• always_naive, always_multi_hop, always_corrective: each fixed strategy applied to every question, regardless of what that question actually needs.
• routed: this course's classifier picks a strategy per question, the same ROUTING_TABLE from Lessons 4-6.

For each condition, this lesson reports precision@k (the fraction of questions whose expected source document(s) all appear in what got retrieved, extended from naive_rag Lesson 17's single-source version to handle multi-hop questions with two expected sources) and cost (total embedding + generation calls made across all nine questions), so "did routing help" and "what did it cost" are answered together instead of routing's calibration overhead being invisible.

The code, piece by piece

def evaluate_condition(name, strategy_fn, store, query_vectors):
    for (question, expected_sources), query_vector in zip(LABELED_QUESTIONS, query_vectors):
        retrieved, embed_calls, generate_calls = strategy_fn(question, query_vector, store)
        hits += set(expected_sources).issubset({r["source"] for r in retrieved})
        ...

Every strategy function (naive_strategy, multi_hop_strategy, corrective_strategy) returns not just what it retrieved but how many extra embed/generate calls it made beyond the one query embedding every condition already shares, so cost is measured honestly per strategy rather than assumed equal.

def evaluate_routed(store, query_vectors):
    for (question, expected_sources), query_vector in zip(LABELED_QUESTIONS, query_vectors):
        label = classify(question)
        total_generate += 1  # the classify call itself
        strategy_fn = ROUTING_TABLE[label]
        retrieved, embed_calls, generate_calls = strategy_fn(question, query_vector, store)
        ...

evaluate_routed() is nearly identical to evaluate_condition(), with one addition: it pays for a classify() call on every question, and that call's cost is counted, not hidden. Routing isn't free, this lesson's whole point is showing what it actually buys for that price.

Why the routed-vs-fixed numbers need a caveat

naive_rag Lesson 17's "Why this doesn't generalize (yet)" section already covers the core problem: nine labeled questions are enough to demonstrate the mechanism of an evaluation, not enough to trust the number, and tuning a hyperparameter against the same set you report a score on invalidates that score. Read that section if you haven't; it isn't re-derived here.

This course is the series' clearest case of exactly that second trap. ROUTING_TABLE's mapping (Lesson 4-6) and CLASSIFY_PROMPT's wording (Lesson 3) were both adjusted by hand, more than once, specifically against this same nine-question set, until routed matched always_multi_hop instead of losing to it. That is not a held-out result. It's a demonstration that a router can be tuned to match a strong fixed baseline on a small set, not a claim that this routing table generalizes to questions it hasn't seen. A larger, held-out question set, one never looked at while writing CLASSIFY_PROMPT or ROUTING_TABLE, would be needed before "routing helps" could be trusted as a real result instead of a fitted one.

There's a second, more mundane wrinkle worth naming honestly: classify() and grade_chunk()'s grading step call the model with temperature=0, which reduces but doesn't eliminate run-to-run variation, a borderline question near a label boundary (Lesson 16's whole subject) can classify differently between two runs of the exact same script. That's not a bug in this lesson's code, it's a real property of using an LLM as the classifier, and it's one more reason a nine-question score should be read as "here's roughly what happened," not as a precise, reproducible measurement.