Your Vendor Says the AI Is 95% Accurate. What Should You Ask Next?

Ask what it was measured on. A single accuracy number is meaningless without four things attached: the test set, the date, the baseline, and how questions the system should refuse were counted. "95% accurate" can describe a genuinely strong system or a demo tested on forty questions the vendor wrote themselves. Both produce the same number. The questions below separate them, and none require technical knowledge to ask. If a vendor can't answer them, that itself is the answer, because any team that measured properly kept the records.

Why is a single accuracy number not enough?

Because accuracy is a ratio, and the vendor chose the denominator.

Change the test set and the number moves. Test on the queries you designed the system around and it looks excellent. Test on what real users actually type, including the messy and out of scope ones, and it drops.

Neither number is a lie. They answer different questions. The one you care about is the second, and it's rarely the one quoted.

There's also a category problem. In a retrieval system, "accurate" might refer to whether the right document was found, whether the answer was factually correct, or whether the citation pointed at the right source. These fail independently, and citation accuracy is the hardest of the three. A system can find the right document and still cite the wrong one.

What are the seven questions to ask?

#	Question	What a good answer sounds like	What a bad answer sounds like
1	What was it measured on?	"A 120 query set drawn from real user logs"	"Our internal testing"
2	Who wrote the test questions?	"Sampled from production traffic, labelled by a domain expert"	"Our team put together some examples"
3	When was it last measured?	A specific recent date, with the model version	"That's from the original build"
4	What's the baseline?	"Keyword search got 61% on the same set"	No baseline exists
5	How are refusals counted?	"Out of scope questions are scored separately, we get 94% correct refusal"	"Those aren't really failures"
6	What does the 5% look like?	Named failure categories with examples	"Edge cases"
7	Can I see it fail?	Runs your hardest query live, without preparation	Redirects to a prepared demo

Question seven is the most revealing and the least technical. Ask to type your own query, unrehearsed, using a document you supply. A vendor confident in the system will do this readily. Hesitation here tells you more than the other six answers combined.

Why does the refusal question matter so much?

Because it's where most inflated numbers come from.

A system tested only on questions it should answer never gets scored on questions it should decline. In production, users ask both. A system that confidently answers a question outside its knowledge is producing a wrong answer that no accuracy test caught.

Ask it this way: "Of your test set, how many questions was the correct response to say I don't know?" If the answer is zero, the number describes a friendlier world than the one your users live in.

This matters most in regulated or high stakes settings. A wrong answer about a refund policy is annoying. A wrong answer about a dosage, a contract term, or a compliance rule is a different category of problem.

What should you ask about the 5% that fails?

A vendor who has genuinely measured knows what the failures look like, because someone read them.

Useful follow ups:

• Are failures spread evenly, or concentrated in one query type?
• Do multi part questions fail more than single fact ones? They usually do.
• Does failure look like a wrong answer, or a refusal to answer?
• Which failures are fixable, and which are limits of the approach? The distinction in the third point matters more than it sounds. A system that says "I don't have that information" when it fails is safe. A system that invents a plausible answer when it fails is a liability, even at the same accuracy number. Ask which one you're buying.

How do you verify any of this without technical staff?

You don't need to audit the model. You need to test the claim.

1. Supply your own documents. Not the vendor's demo corpus. Something real, including your awkward formatting.
2. Write 20 questions before the demo. Include five you know are out of scope, and three that need combining two documents.
3. Ask for citations on every answer. Then click through on three of them. Does the cited source actually say that?
4. Ask the same question twice, worded differently. Inconsistent answers to the same question indicate a retrieval problem.
5. Ask something answerable only by a document you deliberately withheld. The correct behaviour is to decline. That fifth test is the single most informative thing a non technical buyer can run. It takes two minutes and it directly measures whether the system improvises.

If you want to go further than a demo, the 48-hour audit checklist covers what a structured inspection looks like across all seven layers of a retrieval system.

What vocabulary should you use in a vendor conversation?

Term	What it means, plainly
Test set / golden set	The fixed list of questions used to measure quality
Baseline	What the number was before, or what a simpler approach achieves. Without it, an improvement claim is unverifiable
Out of scope handling	Whether the system correctly declines questions it can't answer
Hallucination	A confident, fluent answer that isn't supported by any source
Grounding	Each claim being traceable to a real document
Citation accuracy	Whether the source named beside a claim actually supports it
Precision	Of the answers given, how many were right
Recall	Of the answers that should have been found, how many were
Drift	Quality falling over time without anyone changing the code
Eval harness	The automated system that re-measures quality on every change

Using two or three of these correctly changes the conversation. It signals you'll check the work, which tends to produce more careful answers.

When this doesn't apply

Not every situation calls for this level of scrutiny.

• Low stakes internal tools. A meeting notes summariser doesn't need an eval protocol. Try it for a week and judge by whether people keep using it.
• Small pilots with a cheap exit. If the commitment is a month and a small budget, running the pilot is faster than interrogating the claim.
• Genuinely creative features. Draft generation and brainstorming have no correct answer, so accuracy is the wrong frame. Ask about consistency and controllability instead.
• Early stage vendors. A young company may have no formal eval yet. That's a risk to price, not automatically a disqualifier, as long as they say so rather than inventing a figure.
• When you're the one being asked. These questions cut both ways. If you can't answer them about your own system, build the eval harness before the next conversation, because that's the gap to close first.

FAQ

Is 95% accuracy good for an AI system? There's no universal answer, because it depends entirely on the test set and the cost of an error. 95% on real, messy production queries including out of scope ones is strong. 95% on questions the vendor wrote about documents they chose is close to meaningless. The number matters far less than what produced it.

What if the vendor won't share their test set? Ask for the structure instead: how many questions, where they came from, who labelled them, and how the categories break down. Legitimate confidentiality concerns don't prevent describing methodology. A refusal to describe even the shape of the test usually means it isn't formal enough to describe.

Should I run my own test before signing? Yes, and it's the highest value hour you'll spend. Twenty questions of your own, on your own documents, including deliberate out of scope ones. This costs nothing and reveals more than any specification document, because it tests the system you'd actually be buying rather than the one that was demoed.

How often should accuracy be re-measured after launch? On every meaningful change, and at least monthly in production. Quality drifts as the document corpus grows and as real user queries diverge from what was originally tested. A number from six months ago describes a system that no longer exists in that form.

What's the difference between accuracy and reliability here? Accuracy is how often the answer is right. Reliability is how predictably it behaves, including whether it fails safely. A system that's 90% accurate and always declines when unsure is often more useful than one that's 95% accurate and invents answers in the remaining 5%.

Who should own this conversation on my side? Whoever owns the definition of a correct answer, usually a domain lead rather than an engineer. Engineering can verify how it was measured, but only someone who knows the subject can judge whether the test questions represent real use. Splitting it this way stops the test drifting toward what's easy to pass.

---

Related reading

• What actually breaks in a RAG system? A 48-hour audit checklist — what a structured inspection covers
• How do you know your AI feature is actually working? — building the measurement yourself
• Why is citation accuracy harder than retrieval accuracy in RAG? — why one accuracy number hides three
• Your AI answers confidently and wrongly: a diagnostic checklist — what the failing 5% usually is
• How to choose an AI development agency: a founder's vetting checklist — the wider vendor conversation
• Build vs buy: should your SaaS build AI in-house or hire an agency? — the decision before the vendor call
• How to measure AI agent ROI — the business number behind the accuracy number