How Do You Know Your AI Feature Is Actually Working?

You know an AI feature works when three things exist: a golden set of 60–120 real queries with expected outcomes, a pass threshold agreed with the client before the build started, and an automated run that executes on every deploy and blocks the release when the threshold is missed. Without all three, "it works" means someone tried a few prompts and liked the answers. Most AI features that fail in production did not fail because the model was wrong — they failed because nobody could tell when the behaviour changed. The eval harness ships before the feature, not after it.

What are the five layers of an AI evaluation harness?

An eval harness is not one test suite. It is five layers with different cadences and different owners.

Layer	What it checks	Cadence	Blocks deploy?
Component eval	Retrieval quality, classification accuracy, extraction correctness — each stage in isolation	Every commit	Yes
Task eval	End-to-end answer correctness against the golden set	Every deploy	Yes
Regression eval	Previously-fixed failures staying fixed	Every deploy	Yes
Guardrail eval	Refusals, PII leakage, out-of-scope answering, prompt injection	Every deploy	Yes
Cost & latency	p50/p95 latency, tokens per request, cost per request	Nightly	Alert only

The common mistake is building only the task layer. When the end-to-end number drops, you then have no way to tell whether retrieval, the prompt, or the model version caused it — so debugging becomes guesswork across three surfaces at once.

A healthy evaluation pipeline follows the same order as the application itself: components are tested first, then complete tasks, then previously-fixed failures, followed by safety checks, and finally operational metrics like latency and cost. This ordering makes failures easier to localise because only one layer changes at a time.

Example CI evaluation run

Stage	Threshold	Result
Component evals	≥95%	✅ 96.8%
Task eval	≥90%	✅ 92.6%
Regression suite	0 failures	✅ 0
Guardrail eval	≥98%	✅ 99.1%
p95 latency	≤2.0 s	✅ 1.9 s

Overall release status: PASS

Notice that the deployment is approved only because every required evaluation layer passed. A strong task accuracy score alone would not permit release if the regression or guardrail evaluations failed.

How do you build a golden set that's actually useful?

The golden set is the asset. Everything else is plumbing.

Everything downstream, including thresholds, regression testing, release gates, and model comparisons, depends on the quality of this dataset.

Source it from real queries, not imagination. Ship a logging layer in week one. After two weeks of internal or pilot usage, sample from what people actually asked. Invented test queries are systematically cleaner than real ones — real users are terse, misspell things, and ask two questions in one sentence.

Size it at 60–120 cases for the first release. Below roughly 60, a single fixed case moves the score enough to be misleading. Above 120, annotation and maintenance cost outruns the marginal signal until the product is mature.

Stratify it deliberately. A useful default split:

Slice	Share	Why it's there
Common happy path	40%	Protects the main use case
Multi-hop / multi-document	20%	Where end-to-end systems degrade first
Out-of-scope	15%	The system must decline, not improvise
Ambiguous / underspecified	15%	Should clarify rather than guess
Known past failures	10%	Grows over time; this becomes the regression suite

Query

Can contractors access customer invoices?

Expected outcome

• Retrieve Access Policy v4
• Mention contractors require explicit approval
• Mention invoice access is audit logged
• Refuse to speculate beyond documented policy

The expected wording is intentionally not specified.

Example golden set

• 75 production queries
• 5 evaluation slices
• 684 annotation units
• 2 human annotators
• Inter-annotator agreement (Cohen's κ): 0.87
• Updated monthly with new production failures

What thresholds should you agree with the client before building?

Agreeing thresholds after the build is a negotiation. Agreeing them before is a specification. Set them in the scoping call and write them into the statement of work.

Dimension	What to agree	Reasonable starting position
Task accuracy	Minimum pass rate on the golden set	≥90%
Out-of-scope handling	Rate of correct refusal	≥98%
Latency	p95, not average	≤2.0 s
Cost per request	Ceiling at expected volume	≤$0.025
Regression	Tolerance for previously-fixed failures	0

The threshold conversation is also the most valuable sales conversation in the project. It forces the client to state what "good" means in numbers, which surfaces mismatched expectations in week one instead of week nine.

Example project outcomes

The table below illustrates how an evaluation harness changes delivery outcomes over the course of a typical AI implementation.

Metric	Before eval harness	Release-ready
Task accuracy (golden set)	74.8%	92.6%
Regression failures	17	0
Out-of-scope handling	81.3%	98.4%
p95 latency	2.8 s	1.9 s
Cost per request	$0.031	$0.024

The largest improvement was not the model itself. It was making quality measurable. Once every deployment ran against the same golden set with a fixed release threshold, regressions became visible before users saw them.

Notice that the largest gain came from preventing regressions rather than improving the underlying model. An evaluation harness primarily increases delivery confidence, not model capability.

What does the audit look like when a client has no evals?

This is the common case. A working diagnostic sequence:

1. Is there a logging layer? No logs means no golden set is possible and no historical comparison exists. This is always the first fix.
2. Can anyone state the current accuracy number? If the answer is a feeling rather than a figure, there is no baseline and no way to prove any change helped.
3. What happened at the last model or prompt change? If nobody knows whether quality moved, the system is being changed blind.
4. Is there a documented out-of-scope policy? Most production incidents in LLM features are confident answers to questions the system was never meant to handle.
5. Who signs off on a release? If the answer is "whoever deployed it," the release gate does not exist.

Each of these maps to a scoped, quotable fix. That is what makes the eval conversation a commercial one rather than a purely technical one.

What vocabulary should you use in an AI system audit?

Term	Working definition
Golden set	A fixed, versioned set of queries with expected outcomes used as the reference for quality
Eval harness	The automated system that runs evals and gates releases
Baseline	The measured quality number before a change; without it, no improvement is provable
Regression suite	Previously-fixed failures kept as permanent tests
Release gate	The threshold that must be met for a deploy to proceed
Component eval	Evaluation of one stage in isolation, before end-to-end
Out-of-scope handling	Correctly declining questions the system was not built to answer
Drift	Degradation in output quality over time without any code change
LLM-as-judge	Using a model to grade outputs, calibrated against human labels
Annotation protocol	The written rules by which humans grade an output, ensuring two people score the same thing the same way

When this doesn't apply

• Throwaway prototypes and demos. If the artefact will not survive the pitch, a harness is overhead. Build it when the thing becomes real.
• Deterministic pipelines with no generation step. Classical software testing already covers this; an LLM-shaped eval adds nothing.
• Purely creative or open-ended features — ideation, brainstorming, draft generation. There is no correct answer to score against, and preference testing with users is the honest substitute.
• Very low volume, human-in-the-loop workflows. When a person reviews every output before it reaches anyone, the reviewer is the eval layer, and formalising it can be premature.
• Before you have real usage data. A golden set built entirely from imagined queries can be worse than none, because it produces confident numbers about a distribution your users do not have.

FAQ

How long does it take to build an eval harness? For a typical retrieval-backed feature, the first working harness — logging, a 60–120 case golden set, a runner, and a CI gate — is a few days of focused work, with golden set annotation the largest share. It is cheaper to build alongside the feature than retrofitted, because the queries and expected outcomes are being decided anyway during the build. In practice, the initial harness is usually built in 2–5 engineering days, with golden set annotation accounting for most of that effort.

Can I use an LLM to grade the outputs instead of humans? Yes, for scale, but only after calibration. Grade 40–60 cases by hand, run the same cases through the judge, and measure agreement. In one illustrative evaluation, the judge achieved 94.8% agreement with human labels after calibration, compared with 81.5% before calibration. If the judge disagrees materially with human labels, the judge prompt is the problem, not the system under test. Recalibrate whenever the judge model version changes.

What's the difference between evals and monitoring? Evals run against a fixed golden set to answer "did this change break something." Monitoring runs against live traffic to answer "is something breaking right now." Evals gate deploys; monitoring triggers alerts. Both are required — a system with only monitoring detects problems after users do.

How often should the golden set be updated? Add every production failure as a case immediately — that is what builds the regression suite. Beyond that, resample from live queries roughly monthly, because real query distribution drifts as users learn what the system can do. A golden set left untouched for six months measures a product that no longer exists.

Isn't this overkill for a small AI feature? The harness scales down. For a small feature, 30 cases in a spreadsheet and a script that runs them is a legitimate eval harness. What does not scale down is having no baseline at all — that is what makes every subsequent quality question unanswerable, regardless of feature size.

Who should own the eval harness on the client side? Whoever owns the definition of a correct answer, which is usually a product or domain lead rather than an engineer. Engineering owns the runner and the gate; the domain owner owns the golden set and the thresholds. Splitting it this way is what keeps the set from drifting toward what is easy to pass.