What this is

No new concepts in this lesson. This is a checkpoint: a small, real system built entirely out of ideas from Lessons 11 through 18, combined into one thing. If you can read the code and explain why the delegation, the output union, and the evaluator are each there, you've mastered the Intermediate tier. If any piece feels unfamiliar, revisit the lesson it came from before continuing to Advanced.

What it does

A support-ticket triage agent that delegates to one of two specialist agents, billing or technical, each returning its own validated response type, plus a pydantic_evals dataset that checks whether triage routes messages correctly.

Where each piece came from

class BillingResponse(BaseModel):
    kind: str = "billing"
    message: str




class TechnicalResponse(BaseModel):
    kind: str = "technical"
    message: str




billing_agent = Agent(
    "google:gemini-3.5-flash-lite",
    output_type=BillingResponse,
    system_prompt="You handle billing questions. Respond briefly.",
)
technical_agent = Agent(
    "google:gemini-3.5-flash-lite",
    output_type=TechnicalResponse,
    system_prompt="You handle technical support questions. Respond briefly.",
)


triage_agent = Agent(
    "google:gemini-3.5-flash-lite",
    output_type=BillingResponse | TechnicalResponse,
    system_prompt=(
        "Route the user's support message to handle_billing or "
        "handle_technical, whichever fits, and return its result."
    ),
)




@triage_agent.tool
def handle_billing(ctx: RunContext[None], message: str) -> BillingResponse:
    """Handle a billing-related support message."""
    return billing_agent.run_sync(message, usage=ctx.usage).output




@triage_agent.tool
def handle_technical(ctx: RunContext[None], message: str) -> TechnicalResponse:
    """Handle a technical support message."""
    return technical_agent.run_sync(message, usage=ctx.usage).output

Lesson 12: the triage agent's tools call billing_agent and technical_agent directly, one agent's tool secretly being a whole other LLM call. Lesson 13: each delegated call passes usage=ctx.usage so the combined cost is visible on the outer result. Lesson 16: the triage agent's output_type is BillingResponse | TechnicalResponse, whichever specialist actually handled it.

Lesson 15 used a built-in evaluator, EqualsExpected. Here, the check is "did triage route to the right specialist," which needs a small custom one, a plain dataclass subclassing Evaluator.

@dataclass
class MatchesKind(Evaluator):
    def evaluate(self, ctx: EvaluatorContext) -> bool:
        return ctx.output == ctx.expected_output




dataset = Dataset(
    name="support_triage",
    cases=[
        Case(
            name="overcharge",
            inputs="I was charged twice for my subscription this month.",
            expected_output="billing",
        ),
        Case(
            name="crash",
            inputs="The app crashes every time I try to open it.",
            expected_output="technical",
        ),
        Case(
            name="refund",
            inputs="How do I get a refund for last week's payment?",
            expected_output="billing",
        ),
    ],
    evaluators=[MatchesKind()],
)

Lesson 15: ctx.output here is whatever the task function returns for a case, this lesson's task returns just the .kind string, "billing" or "technical," to keep the comparison simple; ctx.expected_output is the Case's expected_output. Any comparison logic you want, fuzzy matching, checking multiple fields, calling another LLM as a judge, goes in this one method.

Checkpoint

  • A tool function calling other_agent.run_sync(...) is enough for multi-agent delegation, no special "multi-agent" API needed.
  • What comes back on a Union output type run is determined entirely by which tool the model actually called.
  • Usage passed through delegated calls, usage=ctx.usage, is what makes the true cost of a multi-agent system visible on the outer result.
  • A custom Evaluator subclass is just a dataclass with an evaluate method comparing ctx.output to ctx.expected_output.

Try this yourself, without looking anything up: add a third specialist agent and a third routing tool, then add a case to the dataset that should route to it, and confirm the eval report shows it passing.

If this ran cleanly and made sense without looking anything up, you're ready for the Advanced tier, starting at Lesson 20, where an agent stops running unattended and starts pausing for a human's approval before anything destructive happens.