Tech AI News - 테카이

11. What Happens When You Cite AI-Fabricated Case Law As-Is

· 6 min read

This post was translated from the Korean original by AI.한국어 원문 읽기 →

More lawyers are getting sanctioned every year for filing AI-fabricated case law with courts. What courts punish isn't the fact that 'AI was used' — it's the fact that the result was submitted without being checked.


Background — Why This Topic, Why Now

In June 2023, New York federal judge P. Kevin Castel fined lawyers Steven Schwartz and Peter LoDuca $5,000 each. They had submitted several ChatGPT-fabricated airline lawsuit cases to the court as-is. (ABA Journal) This case, Mata v. Avianca, is often called the starting point of the 'AI hallucinated case law' problem — but it was far from the end.

A database of AI hallucination cases run by HEC Paris researcher Damien Charlotin collects only cases where courts confirmed that AI-fabricated content was actually cited. This tally grew from about 200 cases in mid-2025 to 1,600 cases across 35 countries by June 2026 — nearly an eightfold increase in a single year. (Malay Mail) This isn't happening because lawyers don't know better. AI hallucination is already a widely known risk, yet the same mistake keeps repeating.

The same pattern shows up outside law, too — from fake citations in academic papers to fake book recommendations in news articles. The structure is identical: AI plausibly fabricated something, and a human sent it out without checking.

Key Data & Current State

FieldCaseOutcome / ActionDate
Law (US)Mata v. AviancaTwo lawyers fined $5,000 each2023.06
Law (US)Johnson v. Dunn (N.D. Ala.)Three law firm lawyers publicly reprimanded, barred from the case, referred to state bar for discipline2025.07
Law (worldwide)AI hallucination case database (Charlotin, crowdsourced)1,600 cumulative cases across 35 countries2026.06
Academic publishingLancet analysis of PubMed-indexed papers1 in 277 papers contained fabricated citations (sharp rise from 1 in 2,828 in 2023)2026.05
MediaChicago Sun-Times summer reading list feature10 of 15 recommended books confirmed not to exist; feature pulled and apologized for2025.05

Interpretation: Law, academia, and media all converge on the same failure — "the AI's output was plausible, so the check got skipped." The penalties and responses differ by field, but the cause is one and the same.

In-Depth Analysis

1) The Line for Punishment Is 'Didn't Verify,' Not 'Used AI'

In Johnson v. Dunn, Judge Anna Manasco sanctioned three lawyers who had filed a brief containing five ChatGPT-fabricated citations — a public reprimand, removal from the case, and even a referral to the state bar for discipline. This happened even though the firm already had its own AI usage policy and warning procedures in place. (EDRM) The court didn't take issue with the lawyers using AI tools themselves. What it cited was their failure to fulfill the duty to verify the truth of what was in the brief — the court characterized this as "recklessness in failing to verify the truthfulness of representations."

This standard connects to the signature-responsibility principle under Rule 11 of the US Federal Rules of Civil Procedure. By signing, a lawyer certifies that they have reasonably investigated the content of every brief filed with the court. Whether AI drafted it or a person wrote it by hand, the responsibility borne by the signer is the same.

2) Why It Keeps Happening — The Trap of 'Plausibility'

AI-fabricated case law comes dressed in the exact same format as real cases — case name, year, court, citation style, all present. Only the content is fake; the format is flawless, so a routine skim doesn't catch it. Charlotin's database is heavily mixed with cases from pro se litigants representing themselves, but it also steadily includes cases filed by lawyers. That means even experts with legal knowledge keep making the same mistake.

The more fundamental problem is that the actual cost of verification isn't high at all. Whether a single case actually exists can be confirmed within minutes just by searching the case name and citation number in an official case law database. This is exactly why courts keep demanding it not happen again — not because verification is hard, but because the verification step itself was skipped entirely.

The same structural problem is growing in medical and scientific journals too. An analysis published in The Lancet reviewed roughly 2.5 million PubMed-indexed papers and reported that the share of papers containing fabricated citations had risen more than tenfold in 2026 compared to 2023. (Retraction Watch) One journal cross-checked all 117 footnotes in a published paper, confirmed 6 were AI-fabricated citations, and even discovered that all of the fake citations had been added after the initial peer review — leading it to retract the paper.

Media had a similar incident. A summer reading list feature published by the Chicago Sun-Times in 2025 included 10 nonexistent books out of 15 recommendations. The article had been outsourced by an external content provider to a freelance writer, who admitted to passing along an AI-generated list without verifying it. (404 Media) The structure of the incident is the same across all three fields: the person who ultimately put their name on AI-generated output never checked it.

4) Why 'Failure to Verify' Gets Punished

The reason professions like lawyer, researcher, and journalist exist at all is that the person vouches for the accuracy of their own output. AI cannot stand in for that guarantee. What AI produces is still just a draft, and the moment someone puts their name on it, responsibility shifts entirely to that person.

Analysis: Courts and journals asking "did you verify it" rather than "did you use AI" amounts, in the end, to a procedure for reconfirming who holds that guarantee of responsibility. How much of the work AI took on doesn't change where the responsibility lies.

In Practice — Don't Cite It Until You've Verified the Source Yourself

  1. Treat any citation, figure, or case AI gives you as 'nonexistent' until you've opened the original source and confirmed it yourself. No exceptions, even when the format looks correct.
  2. Even if the citation is real, you're not done. Check the original source directly to confirm it actually supports what you're claiming. There are sanctioned cases where a real case was simply cited in the wrong context.
  3. Don't ask the AI itself, "is this really real?" The same model can fabricate another piece of made-up evidence and answer "yes." Verification must always happen against the primary source.
  4. Don't use anything you don't have time to verify. Don't hedge with phrases like "it is reportedly..." — if you couldn't confirm it, just leave it out.
  5. Establish the principle that whoever puts their name on the output is the final verifier. Even if AI wrote the draft, the person who reviewed and signed off bears responsibility.

References

  • #ai
  • #ai-hallucination
  • #legal-ai
  • #source-verification
  • #academic-misconduct
  • #professional-responsibility
How It Works

Transformer — The Ancestor of Every AI Today

From GPT to Gemini to Claude, every major AI model in 2026 runs on the Transformer architecture Google published in 2017. This article covers how self-attention works and the line of variations from decoder-only models to MoE and Mamba hybrids.

· 5 min