How to Build a Golden Evaluation Dataset for Your AI Agent

Quick answer

A golden evaluation dataset is a fixed, versioned set of real representative inputs (30-100+ depending on task diversity) with expected outputs or a scoring rubric, run against your agent before every deployment — prompt change, model swap, or code change — so quality regressions are caught automatically instead of discovered by users. Build it from real production inputs and edge cases, not synthetic examples, and expand it every time a bug reaches production.

Why "it looked good in a few manual tests" isn't evaluation

Manually trying an agent with a handful of prompts before shipping catches obvious breakage but misses the long tail — the edge cases, ambiguous phrasings, and adversarial inputs that show up once real users start using it. A golden eval set formalizes what "good" means for your specific task and makes checking for it repeatable, so every change gets tested against the same bar instead of a developer's spot-check.

Source real inputs, not synthetic ones

The most valuable eval examples come from actual production traffic or actual buyer questions your team has fielded — real phrasing, real ambiguity, real edge cases — not inputs a developer invents while thinking about the happy path. If you don't have production traffic yet, source from red-teaming sessions, support tickets, or sales call transcripts where the exact question came up. Synthetic examples systematically miss the weird phrasing and edge cases real users actually produce.

Cover the failure modes you already know about, deliberately

Beyond typical inputs, deliberately include known hard cases: ambiguous requests, adversarial prompts, inputs that previously caused hallucinations or errors, and edge cases specific to your domain (a multi-part question, a request outside the agent's scope, a request in a different language if relevant). Every production bug that reaches you should get added to the eval set as a permanent regression test — this is how the set compounds in value over time instead of staying static.

Score against a rubric, not just exact-match

For open-ended agent responses, exact string matching rarely works — build a rubric instead (did it answer the actual question, was it factually grounded in provided context, did it follow required format, did it avoid claims it can't support) and score against that, either manually for a small set or with an LLM-as-judge approach for scale. This connects directly to observability and eval tooling in production, which is the same scoring discipline applied continuously rather than just pre-deployment.

Run the full set before every meaningful change

The eval set only has value if it's actually run before shipping — a prompt tweak, a model swap (see migrating between LLM providers), a tool change, or an underlying model version update from your provider. Wire this into your deployment process so it's not optional or easily skipped under time pressure; a regression caught by an eval run costs a delayed deploy, while one caught by a user costs trust.

Version the eval set alongside your prompts

Track the eval set itself in version control next to your prompts and code, so you can see exactly what "passing" meant at any point in time and compare scores across versions meaningfully. An eval set that silently changes over time makes it impossible to tell whether a score change reflects a real quality shift or a different test.

FAQ

How many examples does a golden eval set need to start being useful? 20-30 well-chosen real examples covering your main task types and a few known edge cases is enough to start catching obvious regressions; expand from there as you learn about new failure modes.

Should eval scoring be automated or manual? Start manual for a small set to build confidence in the rubric, then move to LLM-as-judge scoring for scale once you trust the rubric captures what actually matters — full automation without a validated rubric can miss what a human reviewer would catch.

Does a golden eval set replace manual QA before launch? It replaces repetitive manual re-checking of known cases, but new features or major changes still warrant fresh manual review — the eval set is a regression safety net, not a substitute for judgment on new territory.

How often should the eval set be updated? Continuously — every production bug or edge case that reaches a user should be added as a new eval example, so the set grows more comprehensive over the life of the product.

Accelate builds a golden eval set alongside every agent as a required deliverable, not an afterthought — it's the difference between "we tested it" and "we can prove it still works."