When AI makes things up: how to spot it and how to prevent it
A model does not invent out of malice but because it produces the likely continuation of a text — and an invented study is a very likely continuation. It cannot be prevented entirely, but it can be largely contained.
In short
- Invented details arise not on hard questions but on ones with an expected answer shape — figures, sources, statutory references.
- Five kinds are particularly affected: studies, search volumes, legal references, product details and quotations.
- The most effective measure is a ban on invention in the prompt plus an obligation to mark what is unknown.
- A confident tone is no sign of accuracy — models phrase invented details exactly as assuredly as substantiated ones.
The term "hallucination" misleads, because it sounds like an exceptional state. In fact it is the same process that produces the correct answers: the likely continuation of a text.
Asked for a study on a topic, the likely continuation is a study reference with an institute, a year and a percentage. Whether that study exists is not a variable in the production.
Five kinds of detail that are particularly affected
1. Studies and statistics
"According to a 2024 study, 67 per cent of small businesses use …" — institute, year and percentage work together very convincingly, and all three can be freely invented.
Warning signs: round numbers, no link, an institute with no exact title.
2. Search volumes and market figures
No model has access to a search engine's query data. Ask anyway and figures still appear — because the question expects a figure.
Warning signs: any figure in this area. Without exception.
3. Legal references and deadlines
Article numbers, section numbers, transition periods. Particularly dangerous because they sound authoritative and, in practice, trigger decisions.
Warning signs: a precise number with no mention of the version and date.
4. Product details and prices
Feature scope, tier limits, vendor prices. These change frequently and often sit behind login walls.
Warning signs: exact amounts with no date and no source.
5. Quotations and details about people
Verbatim quotations, job titles, who said what when. The damage is greatest here, because something is attributed to real people.
Warning signs: any quotation without a verifiable location.
Worth knowing
Tone reveals nothing. Models phrase invented details with the same assurance as substantiated ones — a "probably" or "possibly" does not appear where the uncertainty actually sits.
That is why the usual checking method fails: people judge credibility partly by confidence of delivery. With models that signal is absent — you have to check the detail itself, not its presentation.
What actually helps
- A ban on invention in the prompt. The instruction to use no figure and no source that is not in the supplied material — and to mark what is unknown as
[detail missing]. The single most effective measure. - Supply material rather than query it. Write your own figures into the prompt and you need no invented ones. The difference between "write something about X" and "write something about X, here are our figures on it" is the biggest single quality jump there is.
- Search-capable models for anything current. If the model can search and names its sources, details can be verified. But not verifying is not the same as verified.
- Checking as its own step. Have a list of all verifiable claims produced, then check them yourself. Separated from the writing, or it gets skipped.
- Figures with provenance. Every figure in the finished text gets a source note — even if that note reads "experience-based estimate, not a survey".
The most expensive case is not the obviously wrong detail but the one that is narrowly off: a deadline that exists but with a different date. A provision that exists but governs something else. A vendor price that was correct two years ago.
Such details survive any superficial check, because they can be confirmed — just in a different version. Which is why it is not enough to check that the source exists; you have to check that it says what the text says.
The prompt rule, word for word
Three sentences that belong in every prompt where verifiable details can arise:
Rules for details: 1. Use no figure, no study, no legal reference and no quotation that is not present in the material I supplied above. 2. Where you need a detail that is missing there, write [detail missing: which] instead of adding something. 3. Mark every statement you cannot substantiate from the material with [unverified] at the end of the sentence. These three rules take precedence over every other instruction in this prompt.
That last sentence is not redundant: details at the end of a prompt are taken into account more reliably, and an explicit precedence prevents the rules being traded off against the wish for smooth prose.
Checking with a second pass
Check the following text for details that may be invented. Be strict; praise nothing; do not rewrite the text. Text: [paste text] Tasks: 1. List every verifiable claim individually: figures, percentages, studies, legal references, years, product details, prices, quotations. 2. Mark each one: substantiable from the text itself / I have to check it externally / looks invented. For "looks invented", name the warning sign. 3. Name the three details whose falsity would do the most damage, and justify that. 4. For every unsubstantiated detail, phrase a version that works without it and still says something. Do not invent source references or evidence. If you cannot place a detail, say so.
In closing
Invented details are not a fault in the tool but a property of the process. They do not go away, but they can be contained: supply material rather than query it, put a ban on invention in the prompt, and treat checking as its own working step.
The uncomfortable consequence: every verifiable detail in a published text has to have been checked by a person. That is the part that cannot be automated — and the reason subject expertise becomes more important when using language models, not less.
Common questions
Why do language models make things up?
Because they produce the likely continuation of a text. Asked about a study, a study reference with an institute, a year and a percentage is the likely continuation — whether that study exists is not a variable in the production. It is the same process that produces the correct answers.
Which details are particularly affected?
Five kinds: studies and statistics, search volumes and market figures, legal references and deadlines, product details and prices, and quotations and details about people. What they share is that the question expects a particular answer shape — a figure, a section number, a quotation.
Can you spot invented details from the tone?
No. Models phrase invented details with the same assurance as substantiated ones; qualifiers like "probably" do not appear where the uncertainty actually sits. What has to be checked is the detail itself, not its presentation.
Does asking "are you sure?" help?
No. A model can confirm an invented detail within the same conversation or change it under pressure — neither says anything about accuracy. Checking happens at the source, not in the conversation.
What helps most effectively against it?
Supplying material rather than querying it: write your own figures into the prompt and you need no invented ones. Add an explicit ban on invention with an obligation to mark unknowns — at the end of the prompt, taking precedence over all other instructions — and treat checking as its own working step.
Marketing that sets itself up
The Studio Engine beta is live. Claim your spot and help shape it from the start.
Join the beta →