Why the Same AI Agent Gives Two Different Answers to the Same Question (And How to Fix It)

Quick answer

Contradictory answers to the same question usually come from three sources: sampling randomness (temperature), retrieval variance in a RAG pipeline pulling slightly different context each time, or ambiguous phrasing that resolves differently depending on conversation history. Fix it by lowering temperature for factual queries, caching or pinning retrieval results for high-frequency questions, adding a deterministic lookup layer for facts that have one correct answer, and logging repeat-question pairs so you can detect drift before customers do.

This is a different problem than hallucination

Most reliability content about AI agents focuses on whether an answer is right. This post is about something narrower and, in some ways, more damaging to trust: two answers that are each individually plausible, neither obviously wrong on its face, but that contradict each other. A customer asking "what's your return window" and getting "30 days" on Monday and "45 days" on Thursday does not experience that as a hallucination. They experience it as the agent being unreliable, or worse, as being lied to. This is distinct from the hallucination and error-rate problem, where an answer is factually wrong, and from honest uncertainty design, where the agent should express doubt but often confidently states two different "facts" instead.

Where inconsistency actually comes from

Sampling randomness. If your agent runs at a non-zero temperature, phrasing variance is expected and usually harmless. But if the underlying facts diverge, not just the wording, temperature is amplifying an underlying lack of grounding rather than causing the problem outright.

Retrieval variance. In a RAG-backed agent, the same question asked twice can retrieve slightly different chunks if your index has near-duplicate documents, if chunk boundaries split a fact awkwardly, or if a re-ranker's tie-breaking isn't deterministic. The agent isn't being inconsistent; it's answering based on genuinely different context each time.

Conversation-history dependence. The same literal question, asked as the first message versus asked after ten turns of unrelated context, can resolve differently because the model is weighing prior turns that shifted its framing.

Stale vs. fresh knowledge. If part of an answer comes from a fine-tuned or cached response and part comes from live retrieval, the two can drift out of sync over time.

Fixes that actually move the needle

1. Route factual, policy-type questions to a deterministic lookup, not free generation. Return windows, pricing tiers, and business hours have exactly one correct answer at any given moment. Store them as structured data and have the agent retrieve and quote that value rather than generating it from a general sense of the policy. This overlaps with the discipline in building a golden eval dataset: the same fixed-answer questions that belong in your eval set are the ones that most need a deterministic source of truth in production.

2. Lower temperature for factual intents specifically, while leaving conversational, empathetic responses at a higher temperature where variance is a feature, not a bug.

3. Deduplicate and consolidate your retrieval index. Near-duplicate source documents are the most common hidden cause of retrieval-driven inconsistency; run a periodic dedup pass rather than only tuning the retriever.

4. Log question-answer pairs and flag repeats with diverging answers. A nightly job that clusters semantically similar customer questions and checks whether the agent's answers to each cluster agree catches drift before it becomes a support escalation.

5. Cache high-frequency factual answers so the same question, asked minutes apart by different customers, is guaranteed to get the identical response rather than a fresh generation each time.

What not to do

Don't try to solve this purely by lowering temperature to zero across the board. That kills useful variance in tone and empathy, and it does not fix retrieval-driven inconsistency at all, since a deterministic model can still be fed different context on different calls. Treat it as a retrieval and knowledge-architecture problem first, a sampling problem second.

FAQ

Q: Why the Same AI Agent Gives Two Different Answers to the Same Question (And How to Fix It) A: Contradictory answers to the same question usually come from three sources: sampling randomness (temperature), retrieval variance in a RAG pipeline pulling slightly different context each time, or ambiguous phrasing that resolves differently depending on conversation history. Fix it by lowering temperature for factual queries, caching or pinning retrieval results for high-frequency questions, adding a deterministic lookup layer for facts that have one correct answer, and logging repeat-question pairs so you can detect drift before customers do.