Checkpoint: Notes Assistant That Reports Its Own Routing

What this is

No new concepts in this lesson. This is a checkpoint: a small, real script built entirely out of ideas from Lessons 10 through 17, combined into one thing. If you can read lesson.py and understand why every piece is there, you've mastered the Intermediate tier. If any piece feels unfamiliar, that's a sign to revisit the lesson it came from before continuing to Advanced.

What it does

Loads (or builds and persists) an embedding index over every fixture note, then answers three questions, one simple, one multi-hop, one ambiguous, each routed automatically to a different one of the three strategies. Every answer discloses, in its own text, which strategy handled it and why, and every routing decision is logged to a local file for later inspection.

Where each piece came from

def classify_with_confidence(question: str) -> tuple[str, float]:
    ...
    return result["label"], float(result["confidence"])

Lesson 10: the classifier returns a label AND a confidence score, from one structured-output call.

def route(label: str, confidence: float, question: str, store: list[dict]) -> tuple[str, list[dict]]:
    if confidence < CONFIDENCE_THRESHOLD:
        return "multi_hop (fallback)", retrieve(question, store, k=2)
    ...

Lesson 11: below CONFIDENCE_THRESHOLD (0.8, read off Lesson 10's own examples, not tuned against Lesson 17's evaluation set), the label is set aside and retrieval falls back to the wider strategy. None of this checkpoint's three demo questions actually trigger the fallback, each classifies confidently enough to route on its own label, the fallback path itself was already demonstrated in isolation back in Lesson 11.

if label == "ambiguous":
    scored_all = retrieve(question, store, k=len(store))
    top1 = scored_all[:1]
    if grade_chunk(question, top1[0]["text"]) == "relevant":
        return "corrective", top1
    return "corrective", scored_all[:2]

Lesson 12's grading idea folded into the corrective route: grade the top result before trusting it, widen only if the grade says no.

def log_decision(entry: dict) -> None:
    with LOG_PATH.open("a") as f:
        f.write(json.dumps(entry) + "\n")

Lesson 13: every routing decision, question, label, confidence, strategy, sources, and call counts, appended to routing_log.jsonl.

CALL_COUNTS = {"embed": 0, "generate": 0}
def reset_counts() -> None: ...

Lesson 14: call counts reset per question and logged alongside the decision, so cost is visible per answer, not just aggregated.

SYSTEM_PROMPT = """... End every answer with one final line in exactly this form:
Strategy used: <strategy name> (<one-line reason why this strategy fit this question>)"""

Lesson 15: the strategy and the reason for it are generated inside the model's own answer text, not printed separately by this script.

(Lesson 16's misrouting failures and Lesson 17's evaluation don't appear directly in this script's code, since this checkpoint routes three clean, unambiguous demo questions rather than a labeled set built to expose edge cases, but everything those lessons taught about where routing can go wrong, and how honestly to measure whether it helps, is what this script's design decisions rest on.)