Pydantic Evals Basics

The LangSmith equivalent, without leaving your codebase

The LangSmith course built datasets and evaluators against a hosted service: you uploaded examples, ran your chain against them, and read results in the LangSmith UI. pydantic_evals is the same idea, define a set of inputs with expected outputs, run something against them, score the results, but it runs entirely locally as a Python library, no account or API key beyond Gemini's needed.

Case, Dataset, and evaluators

A Case is one example: an input, and, optionally, the output you expect.

from dotenv import load_dotenv
from pydantic_ai import Agent
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import EqualsExpected

load_dotenv()

agent = Agent(
    "google:gemini-3.5-flash-lite",
    system_prompt="Answer with just the capital city name, nothing else.",
)


def ask_capital(country: str) -> str:
    return agent.run_sync(country).output


dataset = Dataset(
    name="capitals",
    cases=[
        Case(name="france", inputs="France", expected_output="Paris"),
        Case(name="japan", inputs="Japan", expected_output="Tokyo"),
        Case(name="egypt", inputs="Egypt", expected_output="Cairo"),
    ],
    evaluators=[EqualsExpected()],
)


def main() -> None:
    report = dataset.evaluate_sync(ask_capital)
    print(report)

EqualsExpected is one of several built-in evaluators, others include Contains, IsInstance, and LLMJudge for cases where "equals" is too strict and you want a model to judge the answer instead. An evaluator's job is always the same shape: look at a case's actual output versus its expected_output, or whatever else it checks, and produce a pass/fail or score.

Running the dataset against your agent

Dataset.evaluate_sync takes any callable from input to output, here, a thin wrapper around agent.run_sync, runs it against every case, and prints a summary table. The report shows one row per case, pass/fail per evaluator, duration, and an averages row. This is the same "did my system regress" question LangSmith's dataset runs answer, just rendered to your terminal instead of a dashboard, which makes it something you can run as part of a test suite, not just a manual check.