The Hallucination Problem: Why ChatGPT Cites Cases That Don’t Exist (And Why Judges Know It)

At Lotus Appellate Law, we have encountered pro se clients who come in with motions they have drafted using ChatGPT. They are proud of them. The language sounded professional. It cited cases. It made arguments that sounded legal. So they file it. Confidently.

Then we run the citations.

Half of them do not exist.

Not “we could not find them.” Not “the case name was slightly off.” The cases do not exist. ChatGPT generated case names and citations that sounded plausible but were completely fabricated.

What AI Hallucinations Look Like

If you ask ChatGPT to help draft a legal motion, it will confidently generate citations to cases. The case names will sound real. The citations will follow proper formatting. The reasoning will be plausible.

And some of them will be completely made up.

This is called a hallucination. The AI is not trying to mislead you. It is not trying to deceive the court. It is generating what seems like plausible legal content. But it is generating text, not retrieving actual cases. When the AI does not have reliable training data about a particular legal issue, it invents citations that sound like they could exist.

So you end up with a motion that cites Smith v. Jones, 123 Utah 456 (2015)—a case that never existed. Or a statute that reads like a real statute but does not actually appear in the Utah Code. Or a rule that sounds correct but does not exist.

And you file it. Confidently. Because the AI was confident.

What Happens When A Judge Runs The Citation

When a judge encounters a citation to a case that does not exist, the response is immediate and unforgiving.

The judge runs the citation. Nothing comes up. The judge knows instantly: either this is a made-up case or the attorney does not know how to cite properly. Either way, it is a red flag.

If the motion came from a law firm, the judge might assume it is a typo and give the attorney an opportunity to clarify. But for a pro se motion? The judge views it as evidence that the filer does not know what they are doing.

And once that judgment is made, the judge does not give the motion a charitable reading. The judge reads it skeptically. The judge looks for reasons to dismiss it. Under URCP Rule 11, the judge can also sanction you for filing frivolous work.

A motion that cites a non-existent case is going to be ruled against. Not just on the merits. On credibility. The court concludes the motion is unreliable.

When AI Misreads Real Cases

But hallucinations are not the only problem. Sometimes AI cites real cases but fundamentally misunderstands what they hold.

You end up with a motion that cites a case for the legal proposition that it actually contradicts. The case name is real. The citation is correct. But the reasoning AI pulled from it is backward. The case stands for the opposite proposition from what the motion claims.

A judge will catch that immediately. When you cite a case for something it does not actually say, that is not a technical error. That is a credibility problem. The judge concludes you either do not understand the case or you are deliberately misrepresenting it.

Either interpretation means the motion gets dismissed. The damage is permanent.

The System Cannot Accommodate This

Here is what matters: judges are overburdened. They do not have time to fact-check every citation in a pro se motion. They do not have time to run citations and verify that the cases are real. They do not have time to distinguish between “you are incompetent” and “your AI made things up.”

So when a judge encounters a hallucinated citation, the judge does not spend time trying to figure out what case you might have meant. The judge concludes: this person does not know the law. This motion is not reliable. This is being dismissed.

The trial court moves on. Your motion is gone. Your issue is waived. Your case is damaged.

If appellate counsel is later brought in to appeal, the dismissal is final. The motion was flawed. The court was right to dismiss it. There is nothing to appeal.

The Credibility Consequence

The worst part is what this does to your credibility with the court. Once a judge sees that your motion cited a case that does not exist, or cited a case for a proposition it contradicts, the judge has reached a conclusion about your case: this is not serious. This person does not understand the law. This is not worth serious consideration.

That credibility damage follows your case through trial and on appeal. If you later file another motion, the judge is already skeptical. If the case goes to trial, the judge has already decided you are not reliable. That prejudgment affects how the judge reads everything you file.

What This Actually Means

When you use AI to draft a legal motion, you are not just betting that the AI will organize the argument well. You are betting that the AI will not hallucinate a citation. That it will not misread a case. That it will not generate something that sounds legal but is not.

And if you lose that bet—if the AI generates a non-existent case or misreads a real one—your motion is gone. Your credibility is gone. Your case is damaged.

Judges do not have time to give you credit for trying. They apply the standards strictly and move on.

---

KEY RULE

AI Hallucinations in Legal Citations—Fatal to Your Motion

ChatGPT generates case citations with complete confidence, including citations to cases that do not exist. When a judge runs a citation and finds nothing, the judge concludes the filer either does not know the law or is unreliable. A motion containing hallucinated citations will be dismissed. The damage to credibility is permanent. Judges do not excuse hallucinated citations as a good-faith mistake; they treat them as evidence of incompetence or frivolousness.

---

WHAT THIS MEANS

The promise of AI is that it can write like a lawyer. But it cannot think like a lawyer. It cannot distinguish between real cases and plausible-sounding fabrications. It cannot understand the difference between citing a case correctly and citing it backward.

When you rely on AI to draft your motion, you are accepting the risk that the AI will hallucinate. And when the AI hallucination lands in front of a judge, the motion is dismissed.

This is why understanding what appellate counsel does matters. Appellate counsel does not just write motions. Appellate counsel verifies citations, checks the law, ensures accuracy. For trial counsel wanting to protect appellate options, trial-stage record review identifies which issues were preserved and which have been waived before pro se mistakes are made.

The time to get representation is before you draft, not after the judge has dismissed your AI-generated motion based on hallucinated citations.

---

Meaningful legal representation means motions that cite real cases, arguments that are legally sound, and filings that the court takes seriously. At Lotus Appellate Law, we ensure that every citation is verified, every case is understood, and every motion complies with Utah law. If you have questions about your case or want to discuss whether your appellate options are protected, the next step is a conversation — schedule a call with Lotus Appellate Law.