Guide
Why AI makes up answers, and what stops it
How next-token prediction produces confident invented facts, why the scoreboard keeps rewarding it, and what that means for what you buy.
Published Updated 8 min read
The short answer
AI makes up answers because it was built to predict the next word, not to look anything up. The model reads your question and produces the continuation that fits best given everything it has seen. Fitting well and being true are different targets. When the true answer is not in the model, the plausible one still is.
That is the whole cause. The model is not lying, because lying means knowing the truth and choosing otherwise. It is not broken, because it is doing exactly the job it was trained to do. It produces a well-shaped answer to a question it cannot answer, and a well-shaped wrong answer looks identical to a well-shaped right one until somebody checks.
Everything else follows: why the invented details are so specific, why they arrive without a hedge, why fake citations are the most common form. The countermeasures live on another page. Grounding, citations, an abstain path and the evals that gate all three are in how to stop AI hallucinations. Evals are tests for a large language model's answers.
What next-token prediction actually does
A model learns language by reading an enormous amount of text and getting better at guessing what comes next. There is no truth label anywhere in that process. As OpenAI's write-up of the research puts it, the model "sees only positive examples of fluent language and must approximate the overall distribution" (Why language models hallucinate). Nothing in the training data is marked false, so nothing teaches the model what false looks like.
That explains a pattern people find odd. Large models almost never misspell a word or leave a bracket unclosed, yet they will confidently give you the wrong date. Spelling and punctuation follow consistent patterns that sharpen with scale. An arbitrary fact follows no pattern at all.
The paper behind that post puts a number on it. The expected hallucination rate of a pretrained model on a class of arbitrary facts is at least the fraction of those facts that appeared exactly once in its training data. In the authors' example, if 20 percent of birthday facts appear once, expect a base model to hallucinate on at least 20 percent of birthday questions (Kalai, Nachum, Vempala and Zhang, September 2025). Rare facts get invented because rarity is precisely what a pattern-matcher cannot compress.
The authors ran the demonstration on themselves. Asked for the title of one author's PhD dissertation, three widely used models returned three different titles, three different years and three different universities. None was correct. Three fluent answers, zero facts, and no outward difference between them and a right one.
Why fabricated citations are the most common form
A citation is close to pure format. Case name, volume, reporter, court, year. Author, title, journal, page range. That shape is regular, it appears constantly in training text, and the model reproduces it perfectly. The specific content behind any given reference is arbitrary and mostly appears once. So the model gets the shape right and invents the content: a thing that looks verifiable and is not.
The best-documented case is a real court filing. In Mata v. Avianca, Inc., 22-cv-1461 in the Southern District of New York, plaintiff's counsel filed a brief citing cases that did not exist. Ordered by the court to produce copies, they filed fake opinions. Judge P. Kevin Castel's Opinion and Order on Sanctions of 22 June 2023 found that the respondents "submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question." Six fabricated decisions were named in the order, and a 5,000 dollar penalty was imposed jointly and severally (docket, entry 54).
The forensic detail is the part worth sitting with. The court found the fake "Varghese" decision carried a real docket number belonging to an unrelated case, and a Federal Reporter citation belonging to a different real decision. Its quotation marks were often unpaired. It ended abruptly with no conclusion. Its legal analysis, the court wrote, "is gibberish." And it cited further cases that also did not exist, each with a citation that resolved to a real but unrelated decision.
The invention was recursive, and every fabricated address borrowed a real, valid-looking one. Nothing in the output signalled invention. Catching it required a person to go and look, which is exactly what did not happen until opposing counsel tried.
Confidence and correctness are separate signals
The certainty you hear in an AI answer is a property of the writing, not a readout of the model's belief.
Consider what the training data contains. Text answering a factual question is overwhelmingly written by someone who knew the answer, so it is specific and unhedged. Text where a person says they are not sure attaches to a different kind of question entirely. Given something that reads like a factual question, the most plausible continuation is a confident specific claim, whether or not the model has anything to base it on. The paper describes the failure the same way: bluffs "are often overconfident and specific, such as September 30 rather than Sometime in autumn."
There is a real internal measure of confidence, called calibration, and the paper reports that base models are often well calibrated after pretraining while post-trained models can drift away from it. But calibration lives in the numbers. Nothing forces it into the prose you read. Your instinct for a person who sounds sure is not merely unhelpful here, it is inverted: the model is most fluent precisely where it has the least to go on, because a rare fact leaves it nothing but the shape of an answer.
The part that is an incentive, not an accident
The cause above explains a base model. It does not explain why the habit survives everything vendors do afterwards. The OpenAI paper's answer: the scoreboard rewards it.
Most benchmarks grade answers right or wrong and give nothing for saying "I don't know." Under that rule a guess is free. It occasionally scores, and abstaining never does. The authors surveyed ten widely used evaluations and found nine of the ten use strict binary grading, and nine of the ten give no credit at all for an admission of uncertainty. The set covers GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH, MuSR, SWE-bench and HLE, and they conclude that the vast majority of popular evaluations grade this way.
What that does to a model shows up cleanly in OpenAI's own published figures for two of its models.
| Metric | gpt-5-thinking-mini | OpenAI o4-mini |
|---|---|---|
| Abstention rate, no specific answer given | 52% | 1% |
| Accuracy rate, higher is better | 22% | 24% |
| Error rate, lower is better | 26% | 75% |
The older model wins on accuracy by two points, which is the single number a leaderboard prints. It is wrong nearly three times as often. A model trained to look good on that scoreboard learns to guess, and guessing is the behaviour you experience as making things up.
The authors' proposed fix is not another hallucination benchmark. It is changing how the dominant benchmarks score, because a good hallucination eval has little effect against hundreds of accuracy-based ones that penalize humility. They are careful about how far the problem goes: "Hallucinations are inevitable only for base models." A system built to answer from a real source and decline otherwise does not have to do this.
What this changes about what you buy
If the cause is that the model answers from patterns rather than sources, no amount of prompt wording fixes it. The fix is architectural. Give the system the data, make it read before it answers, give it a way to decline. Whether you retrieve at answer time or train the knowledge into the model is a real fork, covered in RAG versus fine-tuning. RAG is short for retrieval-augmented generation. Short version: fine-tuning teaches a model form, retrieval gives it facts, so a question about your own documents almost always wants retrieval. Building that layer is what AI agents and grounded RAG is.
None of it is a guarantee. Retrieval moves the failure rather than removing it: hand the model the wrong document and it quotes that one faithfully, in the same confident voice. Grounding and citations do not make mistakes impossible. They make them visible and traceable, instead of silent.
One rule to carry out of this. When an AI system gives you a name, a number, a date or a source, the only useful question is which document it read. If it did not read one, you are looking at the most plausible continuation of your question, and the fact that it sounds certain tells you nothing at all.
How to test this on a system somebody is selling you
You do not need access to the model to find out whether it invents answers. You need three questions and about ten minutes.
Ask it something that is true but absent. Pick a fact that exists in the world and is not in the documents the system was given. A grounded system says it does not have that. A pattern-matcher answers anyway, because a plausible continuation of your question exists whether or not a source does.
Ask it the same question twice, in two sessions. Word it identically. A system reading from a source returns the same value. A system generating from patterns will often return two different numbers, and neither of them is flagged as uncertain.
Ask it where the answer came from, and then open the document. This is the one that separates a real citation from a decorative one. Systems that fabricate references are very good at producing a plausible title, a plausible author and a plausible year. What they cannot do is produce a document that contains the sentence they attributed to it.
If a vendor cannot show you those three tests running on their own system, they have not measured this. That is not the same as their system being bad. It means nobody knows, including them, and you would be buying the answer to a question that has never been asked.
Questions people ask
Why does AI invent citations and sources?
A citation is almost pure format: name, volume, publisher, year, in a highly regular pattern that appears constantly in training text. A model that predicts the next word learns the shape of a reference perfectly, while the specific content behind any one reference is arbitrary and rarely repeated. So it reproduces the shape and fills the content with something that fits. The result reads like a real source and points at nothing, which is why fabricated citations survive a casual glance and fail only when someone looks the reference up.
Why does AI sound so confident when it is wrong?
Confidence in an AI answer is a property of the prose, not a reading of how sure the model is. Training data is full of confident, specific writing and almost none of it is a person hedging about a fact they do not know, so the most plausible continuation of a factual question is a specific claim stated plainly. Kalai and colleagues describe the pattern as bluffing that is overconfident and specific, September 30 rather than sometime in autumn, and that habit is rewarded rather than penalized by the way models are scored.
Why does AI make up quotes that were never said?
A quotation is a shape the model can imitate: attribution, punctuation, register, and subject matter that fits the surrounding argument. Producing text that sounds like a particular person or document is exactly what next-word prediction is good at, and nothing in the process checks whether the sentence was ever uttered. In the Mata v. Avianca sanctions case the court found the submitted fake opinions contained fake quotes and citations, and described the legal reasoning in one of them as gibberish despite its correct outward form.
Is hallucination a bug or is it how the model works?
It is how a bare language model works, and the OpenAI research team argues that makes it fixable at the system level rather than inevitable. Their paper states that hallucinations are inevitable only for base models, since a system can be built to answer from a real source and abstain otherwise. Treat invention as the default behaviour of an unaided model, and treat every countermeasure, retrieval, citations, abstention and evals, as engineering you have to pay for.
Keep reading
- 4 August 2026How to stop AI hallucinationsFour mechanisms that cut invented answers, the number to demand for each, and the eight questions that expose a vendor who has never measured one.
- 4 August 2026RAG vs fine-tuning: what each one costs to ownRAG is short for retrieval-augmented generation. One decision rule, two vendor deprecation clocks you have not planned for, and the two-year cost of owning each approach.