Why AI agents hallucinate — and how to catch it before it reaches a customer

Hallucination — a language model generating content that is plausible, confidently phrased and false — is the risk every AI agent deployment eventually has to answer for. It is tempting to treat it as a bug that the next model release will quietly fix. The research says otherwise: hallucination is a structural property of how these models generate text one likely token at a time, not a defect isolated to any one vendor or version.

A property of how the model works, not an accident

A team from the Hong Kong University of Science and Technology and collaborators, in a 2023 survey published in ACM Computing Surveys, draws a distinction worth keeping: intrinsic hallucination, where the generated text directly contradicts the source it was given, and extrinsic hallucination, where it asserts something the source can neither confirm nor deny. The distinction matters operationally — the first is reduced by better-grounded context (a well-built RAG pipeline, for instance), the second requires an agent that can recognize the edge of what it actually knows, which is a much harder property to engineer.

What actually reduces the risk in production

No single fix eliminates hallucination; what works is layering several. Grounding the agent’s answers in retrieved company documents rather than open memory removes one whole category of intrinsic errors. A visible source citation on every answer turns an invisible failure into a checkable one. And for anything with real consequences — a refund, a contract clause, a regulatory reading — a human stays in the loop before the action is final, not after.

• →A test harness with adversarial, trick questions run before every deployment — not just the happy-path questions the demo was built around.
• →An explicit instruction and reward for saying "I don’t know" when the knowledge base does not contain the answer, rather than filling the gap with a plausible guess.
• →Human sign-off preserved on high-impact decisions — a refund, a cancellation, a regulatory interpretation — even once the agent handles the routine volume unsupervised.

Hallucination will not be engineered away entirely — the research is fairly clear on that. The realistic goal for a production agent is not zero hallucination; it is catching the ones that happen before a customer, an auditor or a regulator does.