Measuring Whether Routing Helps

Where we left off

Lesson 6 finished the three-route router. Every lesson since Lesson 1 has argued, in prose, that routing should do better than always running one fixed strategy. This lesson finally checks that claim against real answers instead of asserting it: a small labeled question set, run both ways, routed and always-naive, scored against what each answer should contain.

A small labeled evaluation set

Seven questions, spanning all three labels this course classifies into: three simple_factual, two multi_hop, two ambiguous. Each one carries what a correct answer needs to contain, a phrase or a small set of phrases:

EVAL_SET = [
    {
        "question": "What oven setting does the pizza dough recipe use?",
        "type": "simple_factual",
        "expect_any_of": ["highest oven"],
    },
    ...
]

is_correct() checks, case-insensitively, whether the answer contains the expected phrase (expect_any_of) or all of a small set of them (expect_all_of, used for the multi-hop and ambiguous questions, where a correct answer genuinely needs to mention facts from more than one document). This is a simple, honest correctness check, not a proper evaluation harness (that's a much bigger subject this series doesn't fully build out until later intermediate and advanced lessons), but it's enough to tell a routed answer that actually contains the right facts from a naive answer that's missing half of them.

Routed vs. always-naive

def answer_always_naive(question: str, store: list[dict]) -> str:
    retrieved = retrieve(question, store, k=1)
    return generate_answer(question, retrieved)

The baseline: naive_rag's exact top-1 shape, run on every question in the set regardless of what type it is, no classification, no widening, no grading. answer_routed() (Lesson 6's answer(), renamed here to line up next to its baseline) is the other column: classify first, then run whichever of the three strategies the label points to.

Where the two should diverge

On the three simple_factual questions, routed and always-naive do the same thing, simple_factual routes to naive top-1 anyway, so no difference is expected there. The interesting rows are multi_hop and ambiguous: always-naive is still stuck at k=1 on those, the exact failure Lesson 1 demonstrated, while routed retrieves more broadly (or grades and retries) specifically because the label told it to. If routing is doing its job, those are the rows where routed should get credit that always-naive doesn't.