Regression Testing in CI

What we're building

The same RAG experiment and pass_rate summary evaluator from Lesson 11, run once, its aggregate score read back, compared against a fixed BASELINE_PASS_RATE, and the script exits with a non-zero status code if the score falls short, exactly the signal a CI system uses to fail a build.

What this reveals

Every experiment so far ended with you reading a score and deciding for yourself whether it looked fine. In a real project, that decision needs to happen automatically, on every pull request, without a human staring at a dashboard each time. Regression testing is running the exact same evaluation Lessons 9-16 already built, but as an automated gate: if the score drops below some agreed baseline, the build fails, the same way a failing unit test blocks a merge.

BASELINE_PASS_RATE here is a stand-in for what would really be a number checked into the repo, representing "the lowest this metric is allowed to be." Raising that number is itself a deliberate decision, made when a real improvement lands, not something this script does on its own.

The code, piece by piece

results = evaluate(target, data=DATASET_NAME, summary_evaluators=[pass_rate], experiment_prefix="rag-ci-check")

The same evaluate() call, and the same pass_rate summary evaluator, from Lesson 11.

project = client.read_project(project_name=results.experiment_name)
summary_feedback = list(client.list_feedback(sessions=[str(project.id)], feedback_key=["pass_rate"]))
current_score = summary_feedback[0].score

An experiment is stored internally as a "project," read_project reads it back to get its id. Per-example evaluator scores attach to individual runs and average out on project.feedback_stats. Summary evaluator scores are different: they attach to the experiment itself, with no individual run behind them (their feedback's run_id is None), so they never show up in feedback_stats. list_feedback, scoped to the experiment's session id and the evaluator's key, is how you read a summary evaluator's own score back.

if current_score < BASELINE_PASS_RATE:
    print("REGRESSION: ...")
    sys.exit(1)

sys.exit(1) is what makes this a real gate rather than just a printed warning: any CI system (GitHub Actions, GitLab CI, anything that runs shell commands) treats a non-zero exit code as a failed step, blocking the pipeline exactly like a failing test would.