Confidence Buckets and Actions

Where we left off

Lesson 3 graded chunks relevant / not-relevant, a deliberate simplification named back in Lesson 1. Lessons 10-11 built the piece that simplification was waiting on, strip-level refinement, somewhere useful for a middle "it's related but doesn't quite answer this" grade to send a chunk. This lesson makes the upgrade: Yan et al. 2024's real three-way grade, correct / ambiguous / incorrect, each mapped to its own action.

Three grades, three actions

• correct: the passage explicitly states the answer. Action: refine (Lesson 11) and keep the result.
• ambiguous: the passage is on the same topic but doesn't explicitly answer the question. Action: refine and keep whatever survives and flag this question as a candidate for rewriting (Lessons 6-7), the grader itself wasn't confident, so a differently-worded retry might turn up something better.
• incorrect: the passage has nothing useful for this question. Action: discard (Lesson 5).

A parsing trap worth knowing about

for bucket in ("incorrect", "ambiguous", "correct"):
    if bucket in grade:
        return bucket

Order matters here: the word "correct" is a literal substring of "incorrect". Checking "correct" before "incorrect" would misclassify every genuinely-incorrect grade as correct, silently. This is a real bug this lesson's own first draft hit, worth knowing about if you ever build a grader with overlapping label names.

The code, piece by piece

AMBIGUOUS_PASSAGE = (
    "The most failure-prone part of the whole setup has been the wind "
    "speed sensor, which occasionally needs attention from time to time."
)

A constructed passage, not a fixture file verbatim, close in vocabulary to the real answer in weather-station.md, but deliberately missing the specific "every few months" schedule. It's on-topic (same sensor, same project) without being a direct answer, exactly the case "ambiguous" exists for.

def act_on_grade(question: str, source: str, chunk_text: str, grade: str) -> str:
    if grade == "correct":
        return f"refine and keep -> {refine(question, chunk_text)!r}"
    if grade == "ambiguous":
        refined = refine(question, chunk_text)
        note = refined if refined else "(nothing survived refinement)"
        return f"refine, keep, AND flag for a possible query rewrite -> {note!r}"
    return "discard entirely"

Three branches, three actions, matching the bullets above.