Why AI makes things up

The short answer

AI makes up answers because it was built to predict the next word, not to look anything up. A language model reads your question and produces the continuation that fits best given everything it has seen. Fitting well and being true are two different targets, and when the true answer is not in the model, the plausible one still is.

That is the entire cause. The model is not lying, because lying means knowing the truth and choosing otherwise. It is not broken, because it is doing precisely the job it was trained to do. It is producing a well-shaped answer to a question it cannot answer, and a well-shaped wrong answer looks exactly like a well-shaped right one until somebody checks.

Everything else follows from that: why the invented details are so specific, why they arrive without a hedge, and why fake citations are the most common form. What to do about it belongs on another page. Grounding, citations, an abstain path, and the evals that gate all three are laid out in how to stop AI hallucinations. This page is the cause, because a buyer who understands the cause stops being surprised and starts asking vendors a better question.

What predicting the next word actually does

A model learns language by reading an enormous amount of text and getting better at guessing what comes next. There is no truth label anywhere in that process. As OpenAI's write-up of the research puts it, the model "sees only positive examples of fluent language and must approximate the overall distribution" (Why language models hallucinate1). Nothing in the training data is marked as false, so nothing teaches the model what false looks like.

That explains a pattern people find odd. Large models almost never misspell a word or leave a bracket unclosed, yet they will confidently give you the wrong date. Spelling and punctuation follow consistent patterns that get sharper with scale. An arbitrary fact does not follow a pattern at all.

The paper behind that post makes the point with a number. The expected hallucination rate of a pretrained model on a class of arbitrary facts is at least the fraction of those facts that appeared exactly once in its training data. In the authors' example, if 20 percent of birthday facts appear once, expect a base model to hallucinate on at least 20 percent of birthday questions (Kalai, Nachum, Vempala and Zhang, September 20252). Rare facts get invented because rarity is precisely what a pattern-matcher cannot compress.

The authors ran the demonstration on themselves. Asked for the title of one author's PhD dissertation, three widely used models returned three different titles, three different years, and three different universities. None was correct. Three fluent answers, zero facts, and no outward difference between them and a right one.

Why fabricated citations are the most common form

A citation is close to pure format. Case name, volume, reporter, court, year. Author, title, journal, page range. That shape is regular, it appears constantly in training text, and the model reproduces it perfectly. The specific content behind any given reference is arbitrary and mostly appears once. So the model gets the shape right and invents the content, which is the worst possible combination: a thing that looks verifiable and is not.

The best-documented case is a real court filing. In Mata v. Avianca, Inc., 22-cv-1461 in the Southern District of New York, plaintiff's counsel filed a brief citing cases that did not exist. Ordered by the court to produce copies, they filed fake opinions. Judge P. Kevin Castel's Opinion and Order on Sanctions of 22 June 2023 found that the respondents "submitted non-existent judicial opinions with fake quotes and citations created by the artificial intelligence tool ChatGPT, then continued to stand by the fake opinions after judicial orders called their existence into question." Six fabricated decisions were named in the order, and a 5,000 dollar penalty was imposed jointly and severally (docket, entry 543).

The forensic detail in that opinion is the part worth sitting with. The court found the fake "Varghese" decision carried a real docket number belonging to an unrelated case, and a Federal Reporter citation belonging to a different real decision. Its quotation marks were often unpaired. It ended abruptly with no conclusion. Its legal analysis, the court wrote, "is gibberish." And it cited further cases that also did not exist, each with a citation that resolved to a real but unrelated decision.

Read that again: the invention was recursive, and every fabricated address borrowed a real, valid-looking one. Nothing in the output signalled invention. Catching it required a person to go and look, which is exactly what did not happen until opposing counsel tried.

Confidence and correctness are separate signals

The certainty you hear in an AI answer is a property of the writing, not a readout of the model's belief.

Think about what the training data contains. Text that answers a factual question is overwhelmingly written by someone who knew the answer, so it is specific and unhedged. Text where a person says they are not sure is attached to a different kind of question entirely. Given a question that reads like a factual one, the most plausible continuation is a confident specific claim, whether or not the model has anything to base it on. The paper describes the failure the same way: bluffs "are often overconfident and specific, such as September 30 rather than Sometime in autumn."

There is a real internal measure of confidence, called calibration, and the paper reports that base models are often well calibrated after pretraining while post-trained models can drift away from it. But calibration lives in the numbers. Nothing forces it into the prose you read. Your instinct for a person who sounds sure is not just unhelpful here, it is inverted: the model is most fluent precisely where it has the least to go on, because a rare fact leaves it nothing but the shape of an answer.

The part that is an incentive, not an accident

The cause above explains a base model. It does not explain why the habit survives everything vendors do afterwards. The answer proposed by the OpenAI paper is that the scoreboard rewards it.

Most benchmarks grade answers as right or wrong and give nothing for saying "I don't know." Under that rule a guess is free. It occasionally scores, and abstaining never does. The authors surveyed ten widely used evaluations and found nine of the ten use strict binary grading, and nine of the ten give no credit at all for an admission of uncertainty. The set covers GPQA, MMLU-Pro, IFEval, Omni-MATH, WildBench, BBH, MATH, MuSR, SWE-bench and HLE, and the conclusion they draw from it is that the vast majority of popular evaluations grade this way.

What that does to a model shows up cleanly in OpenAI's own published figures for two of its models:

The older model wins on accuracy by two points, which is the single number a leaderboard prints. It is wrong nearly three times as often. A model trained to look good on that scoreboard learns to guess, and guessing is the behaviour you experience as making things up.

The fix the authors propose is not another hallucination benchmark. It is changing how the dominant benchmarks score, because a good hallucination eval has little effect against hundreds of accuracy-based ones that penalise humility. They are also careful about how far the problem goes: "Hallucinations are inevitable only for base models." A system built to answer from a real source and decline otherwise does not have to do this.

What this changes about what you buy

If the cause is that the model answers from patterns rather than records, then no amount of prompt wording fixes it. The fix is architectural. Give the system the records, make it read them before it answers, and give it a way to decline. Whether you retrieve those records at answer time or train them into the model is a genuine fork, covered in RAG versus fine-tuning. The short version is that fine-tuning teaches a model form and retrieval gives it facts, so a question about your own documents almost always wants retrieval. Building that layer is what AI agents and grounded RAG is.

None of that is a guarantee. Retrieval moves the failure rather than removing it: hand the model the wrong document and it will quote that one faithfully, in the same confident voice. Grounding and citations do not make mistakes impossible. They make them visible and traceable, instead of silent.

So carry one rule out of this. When an AI system gives you a name, a number, a date, or a source, the only useful question is which record it read. If the answer is that it did not read one, you are looking at the most plausible continuation of your question, and the fact that it sounds certain tells you nothing at all.