Failure Modes of Misrouting

Where we left off

Every lesson so far assumed the classifier gets the label right. It doesn't, always, and this lesson shows what happens on both sides of that failure: a genuinely multi-hop question sent down the cheap naive route, and a genuinely simple question sent down a more expensive route than it needed.

Failure 1: multi-hop, misclassified simple

"What's kept in the same room as the weather station's Raspberry Pi?" needs bookshelf.md and cello-practice.md together, both independently say they're in "the study," the same room the Pi lives in per weather-station.md. Nothing about the question's phrasing names two topics explicitly though, it reads like one factual lookup, and this course's classifier (run for real, not staged) consistently calls it simple_factual. Naive top-1 retrieval only ever returns bookshelf.md, the single closest match, so the answer can only ever name the bookshelf. Cello practice, the second thing that room contains, is invisible to a route that only looked once.

This is the same fundamental gap naive_rag Lesson 16 demonstrated with k=1: a route with no way to know a question needs more than one document has no way to go looking for a second one. Here the failure is one layer earlier, though: it isn't that k=1 was too narrow for a question correctly identified as needing more, it's that the classifier itself never flagged the question as needing more in the first place.

Failure 2: simple, misclassified multi-hop

"What wind speed dries out the garden beds faster?" is fully answerable from garden.md alone, the 20 km/h threshold sits in one sentence there. That same sentence also mentions "the weather station's average readings" in passing, enough shared vocabulary with weather-station.md that the live classifier calls this question multi_hop on some runs and simple_factual on others, the same borderline instability Lesson 10 already measured (confidence around 0.85 on wind-speed questions, right at the edge). This lesson's code pins the label to multi_hop for a repeatable demonstration, but it's a real outcome this classifier actually produces, not a contrived one.

Unlike Failure 1, this isn't wrong: garden.md alone already answers the question correctly, weather-station.md just rides along unnecessarily. The cost isn't a bad answer, it's Lesson 14's cost, an extra chunk and extra tokens spent on a question that never needed them.

Notice the model's own answer in Failure 1 is honest about what it's missing, "the context does not mention what else... is kept in that room", that's Lesson 15's grounded-answer instruction doing its job. The honesty doesn't fix the failure, though: the missing information about cello practice never had a chance to reach the model at all, no amount of careful prompting recovers a document that was never retrieved.