An AI system tells you, with complete assurance, that a particular clause appears in your contract on page four. It does not. Nothing in the answer's tone suggested any uncertainty.
This is called hallucination, and understanding why it happens is necessary before you can design around it — which you can, quite effectively.
It is not a malfunction
The instinct is to treat this as a bug awaiting a fix. It is better understood as a consequence of the mechanism.
The system generates text by producing likely continuations. When it has seen a great deal of relevant material, the likely continuation is usually accurate. When it has not, the mechanism does not stop — there is no component that detects insufficient grounds and halts. It continues producing the most plausible-looking text, and plausible-looking text about a contract clause is a specific-sounding statement about a contract clause.
The fabrication has the same shape as the truth because it was produced by the same process.
Why the confidence
People reasonably expect uncertainty to show. But the hedging in an answer is itself generated text, predicted like everything else. It reflects how such sentences are usually phrased, not an internal assessment of reliability.
So confident phrasing carries no information about correctness. This is the single most important thing to communicate to staff using these systems — a fluent, assured answer is not evidence of anything.
When it is most likely
- Specific factual details — numbers, dates, names, citations, section references. These are exactly where invention is most convincing and most damaging.
- Questions with a false premise. Ask about a clause that does not exist and the system will often describe it rather than contest the premise.
- Niche subjects where little training material existed.
- Long outputs, where accuracy tends to degrade further from the supplied material.
- Anything after the training cutoff, which the system has no access to but will still discuss.
What actually reduces it
Supply the source material. Far and away the most effective measure. Asking a question with the relevant documents attached, and instructing the system to answer only from them, changes the task from recall to reading comprehension — where performance is much better.
Require citations. Ask for the passage each claim came from. This makes checking fast and makes unsupported claims visible.
Permit refusal explicitly. Tell it to say when the answer is not in the material. Without permission, it will produce something.
Keep outputs tight. Narrow, well-scoped tasks hallucinate less than sprawling ones.
Never let it do arithmetic that matters. Hand calculation to a calculator and use the model to decide what to calculate.
What does not work
Asking the system whether it is sure. It will produce a confident reassurance, because that is the likely continuation of that question. Self-reported confidence is not a control.
Instructing it never to make things up. Helpful at the margin, not a solution — it cannot detect the situation reliably enough to comply.
Designing with it
Accept that a residual rate of confident errors is a property of the technology. Then place the system where that is survivable: producing drafts a person checks, handling the clear cases and escalating the rest, working from supplied documents rather than memory.
Used that way, these systems are genuinely valuable. Used as an authority, they will eventually embarrass you.
