The moment AI lies with confidence

Hallucination is not the result of AI 'breaking'. It is closer to a structural side effect: today's training and evaluation methods award more points for a plausible guess than for saying 'I don't know' . Once you understand the mechanism, the way you deal with it changes too.

Background / why this topic now

AI hallucination refers to the phenomenon of a model confidently stating things that are not true as if they were facts. The problem is that this error 'doesn't show'. The sentences are smooth, the format is perfect, and there are even footnotes. It's just that the paper the footnote points to does not exist.

There are two reasons this topic has become important again as of 2026.

First, usage has moved beyond the lab. Generative AI adoption among Korean companies was surveyed at 55.7% (22.4% company-wide plus 33.2% in some departments), and it is projected to expand past 85% in 2026 (Megazone Cloud and Foundry joint survey, 2025). In the same survey, the biggest concern about adoption was precisely **'generation of incorrect information and lack of confidence in results' (61.3%)**.

Second, the explanation of the cause has been academically settled. 'Evaluating large language models for accuracy incentivizes hallucinations' (Kalai et al., 2026), published in Nature in April 2026, defined hallucination not as a defect of the model but as an incentive problem created by evaluation metrics. When the cause changes, the prescription changes.

Key data and current state

Behavior change by evaluation method: On the SimpleQA benchmark, a model that abstained often (gpt-5-thinking-mini) scored 52% abstentions, 22% correct, 26% wrong, while an older model that almost never abstained (o4-mini) scored 1% abstentions, 24% correct, and75% wrong (OpenAI, 2025)
Hallucination rate in summarization tasks: On the Vectara HHEM leaderboard, which measures factual inconsistency in document summarization, top models hallucinate at 1.8–5% (Vectara Hallucination Leaderboard, as of May 2026)
Scale of courtroom damage: More than 1,300 cases worldwide in which courts flagged fake case law or citations generated by AI (AI Hallucination Cases DB, as of August 2026)
Expert diagnosis: 'Hallucination cannot be completely eliminated with current AI technology' — Gartner, publishing eight approaches to minimizing hallucination (January 2026)

? Interpretation: What deserves attention in the figures above is not the third row but the first. Between the model that abstained often and the one that almost never did, the difference in accuracy was only 2 percentage points, but the difference in error rate was 49 percentage points. In other words, 'guessing' barely improved the report card while tripling the risk. And yet benchmarks until now have been designed to favor guessing.

In-depth analysis

1) Why it happens — the fate of a statistical predictor

A large language model is not a device that looks up facts but a device that probabilistically predicts the next word. Using learning theory, the Nature paper showed that for information that does not appear repeatedly in training data (say, a particular person's birthday or the title of an obscure paper), errors at the pretraining stage are statistically unavoidable(Kalai et al., 2026).

One more decisive factor attaches to this. During training, the model does not receive enough negative signals saying 'this is a wrong answer'. Spelling and grammar get corrected because wrong examples are everywhere in the data, but a 'nonexistent court case' does not exist in the data in that form. So the result is born with perfect form and no substance.

2) Why it is 'confident' — the strategy of a good test-taker

This is the heart of the article. On a multiple-choice test, leaving a question blank scores zero, but guessing has an expected value above zero. The accuracy-based benchmarks used to evaluate AI models until now had exactly this structure. Say 'I don't know' and get 0 points, get it wrong and get 0 points, get it right by chance and get 1 point. Then the optimal strategy is to always guess.

OpenAI summed up this incentive by saying that 'accuracy-centered metrics systematically reward guessing over acknowledging uncertainty' (OpenAI, 2025). In other words, the reason AI is confident is not that it is confident, but that it was trained to act confident. There was never any link between the assured tone of a sentence and whether its content is true.

Author's view: As a user, remembering this one sentence gets you halfway to a defense. "AI's tone is not a reliability indicator." Humans evolved to read a hesitant tone as a signal of uncertainty, but LLMs have no such correlation.

3) Does adding RAG solve it? — it reduces, but does not eliminate

The remedy companies use most is RAG (retrieval-augmented generation). By having the model consult internal documents or up-to-date search results before answering, it greatly reduces baseless invention.

But it is no cure-all. According to Korean coverage, RAG operations face limits: ▲ a lack of means to objectively measure quality changes when documents are added or the structure is changed ▲ dozens of tuning parameters such as chunk size, Top-K, embedding model, and prompts ▲ a 'balloon effect' in which improving one area worsens another (ZDNet Korea, 2026). If the retrieved documents are weak or off-topic, the model still fills the gaps with its own imagination.

Gartner, too, in presenting its eight approaches to minimizing hallucination in January 2026, premised them on 'management' tailored to the organization's risk tolerance, not complete elimination .

Practical countermeasures — 7 things users can apply today

① Separate the risk zones first

If you doubt every sentence, there is no point using AI. Hallucination concentrates in specific types.

High risk (always verify): Numbers and statistics, proper nouns such as names of people and places, citations of papers, case law, and statutes, recent events, library and function names in code, quotations
Low risk (usable as is): Polishing sentences, structuring, brainstorming, draft translations, restructuring material I already provided

② State explicitly: 'if you don't know, say so'

If the cause is evaluation incentives, the user can change the incentives directly. Simply appending the following sentence to the prompt induces abstention behavior.

"For anything you are not certain of, do not guess; mark it 'unknown' or 'needs verification'. For each claim, also indicate your confidence (high/medium/low)."

③ Don't ask leading questions

"Tell me the case law on ○○" is a question that plants the premise that such case law exists. The model tries to satisfy that premise. Change it instead to "First check whether case law on ○○ exists, and if not, say so" . The output is contaminated to the extent of the premise planted in the question.

④ Always click the sources yourself

The most dangerous form of hallucination is a lie with a plausible source attached. The classic example is a nonexistent publication assembled from a real journal name, a real author, and a URL in a real format. In Australia, Deloitte refunded part of the fee (about A$97,000) for a roughly A$440,000 report submitted to the government that contained citations of nonexistent papers and fabricated court rulings (October 2025). The presence of a link is not verification. Clicking is verification.

⑤ Paste the original text and work from it

"Summarize this file" is overwhelmingly safer than "Summarize the ○○ paper". The less information the model has to pull from memory, the less room for hallucination. Use modes with search attached, but make a habit of checking the body of any document presented as evidence.

⑥ Cross-check

Pose the same question to a different model and see whether the answers diverge. If two models give different numbers, at least one is wrong, and usually both should be doubted. Asking the same model twice and seeing whether the answer wobbles is also a quick signal.

⑦ Fix where responsibility sits

AI is the drafter, not the final reviewer. For any document going outside, the safest practice is to separate, at the writing stage, 'facts I verified myself' from 'parts AI filled in'.

Implications

That the number one adoption concern among Korean companies is accuracy (61.3%) reads as meaning the bottleneck in Korea is not technology adoption but the absence of a verification process. The tools are already in, but internal rules about who checks the output and by what standard are mostly empty.

The Korean-language environment has one more disadvantage. Since major benchmarks are mostly based on English data, in areas relatively sparse in training data, such as Korean statutes, case law, administrative terminology, and company names, actual errors are likely more frequent than the published hallucination rates. That is why public leaderboard numbers cannot be applied directly to Korean business use.

Author's view: The cheapest defense an organization can build right away is not swapping models but 'AI usage labeling rules' . Simply requiring that documents indicate which parts were written with AI makes the locus of verification responsibility clear.

Outlook and variables to watch

Improvement variable — a shift in evaluation criteria: The Nature paper proposed that switching evaluation to an 'open rubric' approach that penalizes wrong answers and gives partial credit for abstaining would let hallucination-reduction techniques show up in actual scores. If benchmark design changes, model behavior is likely to follow.
Risk variable — agentification: As AI moves from merely answering to executing multiple steps on its own, a hallucination in an intermediate step can be passed as input to the next step and errors can accumulate. The very structure in which humans have fewer chances to see the middle is a risk factor.
Checkpoints: ① Whether major models introduce abstention rates and confidence labeling ② whether Korean-language hallucination evaluation datasets are released for the legal, medical, and financial fields ③ moves by courts and audit bodies to mandate disclosure of AI use

Conclusion

Hallucination is not a bug but the result of incentives. Accuracy-only evaluation has rewarded 'guessing' over 'I don't know', and models simply optimized for that rule.
Tone is not a reliability indicator. There is no correlation between a confident sentence and whether it is true. The presence of a footnote is not verification either.
The goal is management, not complete elimination. Separating risk zones, prompts that induce abstention, attaching the original text, and checking sources yourself: these four alone can prevent most of the incidents individual users run into.

What is AI hallucination?