Tuesday. I changed one line in a support-triage prompt to fix a formatting complaint a customer had flagged twice. Thursday, the agent quietly stopped escalating refund requests over $500. Nobody caught it until a customer had gone nine days without a callback, escalated to a manager, and the manager pulled the ticket history back to that Tuesday.
The change was one word: "concise." I'd added it to stop the agent from padding replies with boilerplate. It complied, and it also decided that dollar thresholds counted as boilerplate worth trimming out of its own reasoning before the routing step ever saw them.
I'd tested the new prompt against three tickets. All three looked cleaner. None of them happened to include a refund over $500.
That's not a prompting failure. It's a testing failure, and it's the one most teams shipping AI agents haven't fixed yet.
The prompt that looked fine in the playground
MIT's NANDA initiative surveyed 300 public AI deployments and interviewed leaders at 52 organizations in 2025. The finding that made headlines: 95% of generative AI pilots show no measurable P&L impact. The finding that actually matters for anyone shipping agents is the one buried under it. The report doesn't blame model capability. It calls the failure a learning gap: tools that don't adapt to a specific workflow, deployed by teams that have no way to measure whether an iteration made things better or worse.
That's a measurement gap wearing a model-quality costume. A team that can't tell you the eval score delta between last week's prompt and this week's prompt isn't iterating. It's re-rolling and hoping the vibes hold.
The playground is where this gap is invisible. You write a prompt, run it against two or three cases you already have in your head, read the output, and it looks right. That's not an eval. That's a glance, and a glance can't catch a failure mode you didn't think to check for: a routing threshold quietly getting classified as filler text.
The fix isn't "write better prompts." The refund-escalation incident showed the prompt itself was fine on the cases it saw. The fix is treating every prompt change the way a merge request treats a code change: nothing ships without running against a suite you didn't write from memory five minutes before you shipped.
Prompts are code. Give them a test suite
OpenAI's own Evals cookbook lays out the mechanism plainly: define an eval once: inputs, expected outputs, grading criteria. Then run two passes through it. A baseline run against the current prompt, and a regression run against the candidate change. Same dataset, same rubric, both scored the same way. The only thing that moves between the two runs is the prompt.
That's the entire trick, and it's almost insultingly simple once you see it stated this way. Most teams already do something like it for code: a test suite that runs on every pull request and blocks the merge if it fails. Almost no team does the equivalent for prompts, because a prompt doesn't look like code. It looks like a paragraph you can just edit and reship.
It is code. It has inputs, it has a specification (the rubric), and it has a failure mode that costs real money when it slips through. Nine days of a customer waiting on a refund is a failure mode with a dollar figure attached.
Anthropic's own guidance on building agents makes the same point from the build side rather than the test side: evals earn their cost when the criteria are clear enough to score and when a model, including the one you're testing, can actually judge whether an answer improved. Build the eval before you optimize the prompt against it, not after something breaks in production. The regression run only catches what the golden set knows to check for, which is why the golden set itself is the asset that matters more than which framework runs it.
A prompt that passes three examples in a chat window hasn't been tested. It's been glanced at.
The judge you're trusting has its own biases
Here's the part that eval vendors don't put on the landing page: the thing scoring your regression run is often another LLM, and that LLM is not a neutral referee.
IBM Research's CALM framework, published at ICLR 2025, quantified 12 distinct biases present in LLM-as-judge scoring across leading models: position bias (the first answer shown tends to win), verbosity bias (longer answers score higher regardless of correctness), and self-enhancement bias (a model rating its own family's outputs more favorably), among others. A separate 2025 study, "Reliability without Validity," ran a large-scale evaluation of LLM judges across 20 NLP tasks and found the same uncomfortable pattern: judges can be internally consistent (they agree with themselves run over run) without being valid. Consistent doesn't mean correct.
Translate that into the refund-escalation incident: if I'd built a golden set and asked an LLM judge to grade "is this response appropriately concise," a verbosity-biased judge would have rewarded the exact failure mode that broke escalation: the shorter, more concise-looking answer that had quietly dropped the dollar threshold. The judge would have told me the regression was an improvement.
This doesn't mean LLM judges are useless. It means the judge is part of the system under test, not outside it. A rubric that only checks tone or length will get gamed by a model optimizing for tone or length - including your own model, during the prompt iteration that's supposed to be making things better. Score for the thing that actually matters (did the routing decision stay correct), not a proxy for it (did the response read as polished).
What actually goes in the golden set
A golden dataset isn't a folder of your favorite five examples. Current eval engineering practice (documented across teams building on Langfuse and similar eval infrastructure in 2026) converges on four buckets, and skipping any one of them leaves a blind spot shaped exactly like the failure you'll eventually ship:
- A stratified sample of real production traffic - not the traffic you remember, the traffic that actually came in, weighted the way it actually distributes.
- An adversarial library - inputs built specifically to break the prompt, not inputs you happened to have lying around.
- Deliberately constructed edge cases - the refund-over-$500 ticket that wasn't in my three test cases because I didn't think to include it.
- Replays of failures that already shipped - every incident becomes a permanent regression test, so the same bug can never reach production twice.
The refund-escalation incident, once it happened, became bucket four. That's the part of this discipline that compounds: a golden set that only grows from bucket one and stays static is dead weight within a quarter, because production traffic drifts and the set stops resembling what's actually coming in. A golden set that absorbs every real incident gets strictly harder to fool over time, which is the entire point of running it at all.
Wiring it into how the team actually ships
None of this works if the eval is a document somebody ran once during a design review. It has to gate the merge the same way a test suite gates a code merge: a prompt change that can't show its score delta against the golden set doesn't go out, full stop.
The 2025 DORA report on AI-assisted software delivery makes the stakes explicit at the org level, not just the prompt level. Its central finding: AI doesn't fix a team, it amplifies what's already there. Teams with strong automated testing and fast feedback loops see AI accelerate them further. Teams without those controls see the same acceleration turn into instability. The report states plainly that AI adoption has a negative relationship with delivery stability specifically where automated testing and version control discipline are weak. Thirty percent of respondents report little or no trust in their own AI-generated code. An untested prompt change is the same failure mode wearing a different hat.
The market is already pricing this in. a16z's 2025 survey of 100 enterprise CIOs found 81% now orchestrate three or more model families in production, up from 68% a year earlier. Procurement has shifted to match, now running through the same rigorous evaluation gates traditional enterprise software always required. A vendor pitching an agent without a reproducible eval score attached is increasingly getting the same skepticism a vendor pitching software without a security review used to get.
What that gate actually looks like in practice is small: a prompt change is a pull request, the pull request triggers a golden-set run, the run produces a score, and the score either clears the threshold or it doesn't.
The judge you didn't design is still grading. It just isn't telling you what it's grading for.
That gate would have caught my one-word edit before it reached a customer. The refund-over-$500 ticket was sitting in the golden set within a week of the incident - bucket four, replayed failure - and it has never regressed again, because the prompt can no longer ship without proving it still handles that exact case correctly.
If you can't tell me what regression suite ran against your last prompt change, you didn't iterate. You gambled, and you haven't lost yet.
Sources
- MIT NANDA - The GenAI Divide: State of AI in Business 2025 - survey of 300 public AI deployments and 52 organizations finding 95% of pilots show no measurable P&L impact, attributed to a workflow-integration learning gap.
- OpenAI Cookbook - Detecting Prompt Regressions with Evals - the baseline-run-vs-modified-run methodology for catching prompt quality regressions before they ship.
- OpenAI - Evals (GitHub) - the open-source reference eval harness and benchmark registry underlying the regression-testing pattern.
- Anthropic - Building Effective Agents - guidance on when evals earn their cost and why they should be built before optimizing agent behavior against them.
- IBM Research - Justice or Prejudice? Quantifying Biases in LLM-as-a-Judge (ICLR 2025) - the CALM framework quantifying 12 distinct biases in LLM-as-judge scoring across leading models.
- arXiv - Reliability without Validity: A Systematic, Large-Scale Evaluation of LLM-as-a-Judge Models (2025) - a 20-task study finding LLM judges can be internally consistent without being valid.
- Google Cloud / DORA - Announcing the 2025 DORA Report - the "AI amplifies what's already there" finding and the negative relationship between AI adoption and delivery stability absent automated testing.
- Andreessen Horowitz - How 100 Enterprise CIOs Are Building and Buying Gen AI in 2025 - survey finding 81% of enterprises now orchestrate 3+ model families in production, up from 68%, alongside more rigorous evaluation-driven procurement.
- Langfuse - Golden Dataset Evaluation: Build and Maintain LLM Test Sets - the four-bucket golden-set composition pattern and the dataset-drift failure mode when sets go unmaintained.
Working through the challenges in this post? I help engineering leaders and CTOs navigate complex technical decisions and scale high-performing teams. Schedule a consultation →