What a Citation Goes Through Before It Reaches You

Lawyers in several countries have now been hauled up in court for filing authorities that an AI made up. The pattern is always the same: a plausible case name, a plausible reporter citation, a paragraph number, a proposition that fits the argument beautifully, and no such judgment anywhere. It reads exactly like the real thing, which is the whole difficulty, because the one person least able to catch it is the person who asked for it.

So when we started building Miss Lucy the question was never how do we make her good at citations. It was what is she not allowed to do, and where do we put the thing that stops her.

We keep a log of every citation the checker has stopped. As of today it holds 676 entries. Here is the breakdown, because the shape of it is more useful than the total:

What was caught	Count
A citation attached to the wrong case	448
Untyped, from before we classified them	206
A citation with no matching judgment at all — fabricated	10
A citation not in the session's own sources	9
A paragraph number that does not exist in that reporter	3

Ten fabrications. Four hundred and forty-eight mismatches. That ratio is the thing worth sitting with, and it is not what most people expect.

The invented case is the easy failure

An invented judgment is catchable, in the end. Search for it and it is not there. A junior can catch it, opposing counsel will certainly catch it, and — this is the point — the machine can catch it too, because "does a judgment by this name exist in the record" is a question with a clean answer.

The mismatch is a different animal. The case is real. The proposition is real. The citation is wrong. Miss Lucy has the right judgment in front of her, knows it reports at, say, AIR 2010 SC 1898, and writes down an SCC citation instead — one that belongs to a different case, or to nothing. Search for the case name and it exists. Read the proposition and it holds. Only when someone pulls the volume does the citation fail to lead anywhere, and by then it is in a filing.

That is the failure mode that actually reaches courtrooms, and 448 of the 676 things our checker has stopped were of that kind.

Where the checking sits

The design decision that matters is an unglamorous one about who is allowed to hold what.

The agents that do the reasoning — the one that researches case law, the one that writes an opinion — cannot see reporter citations at all. They read a citation-free card: the facts, the holding, the paragraphs by the judgment's own internal numbering. They reason on the law and hand back a conclusion. A separate step, which we call the citation desk, is the only part of the system that can pull the formal reporter citations, and it is the only part that writes them.

This is a deliberate narrowing, and it costs us something — it is an extra step, and extra steps are slower. What it buys is that the part of the system doing creative work is structurally incapable of inventing an AIR number, because it never had one.

After the desk writes, five checks run in sequence before a finished document can leave:

1. Fabricated — a citation with no judgment behind it in the record.
2. Mismatched — the record holds an AIR citation for this case and the draft carries an SCC one that does not match it.
3. Unknown — a citation that never appeared in the sources this session actually consulted.
4. Invalid pinpoint — a paragraph number that is not in that reporter's own paragraph map for that judgment.
5. Invalid page-letter anchor — the same test for SCR page letters.

Paragraph numbers are a separate problem, and we solved it by refusing

A reporter pinpoint is harder than a citation, because the paragraph numbering in SCC is not the paragraph numbering in SCR, and neither is the numbering the court itself used. "Paragraph 14" is not a fact about a judgment. It is a fact about an edition of a judgment.

So we built paragraph maps. 34,445 judgments in the record now carry one — 27,132 for SCC, 8,364 for the Supreme Court's own neutral series, 756 for SCR. Where a map exists, a pinpoint can be checked against it and translated between editions.

Where no map exists, the rule is the one-line version of everything above: no paragraph map, no pinpoint. She drops the paragraph number rather than guess it. A citation that says (2010) 11 SCC 441 and stops there is less useful than one that lands you on the paragraph. It is also the only one of the two we can stand behind, and a document that quietly gives you the more useful of those two things has made a decision on your behalf that it had no business making.

What is checked, and what is not

This part gets stated plainly because we would rather say it than have someone find out.

The checks run on the documents she delivers — the opinion, the notice, the petition, the paper book, anything you download or she hands over as a finished artefact. Those go through the citation desk, the five checks, and a separate two-pass verifier that reads the draft back against its own sources before it is released.

An ordinary citation mentioned in the middle of a chat is not put through that pipeline. If you ask her a question in the composer and she mentions a case in her reply, that is her working out loud, at the speed of a conversation, and it has the standing of a conversation. It is a starting point to check, not an authority to rely on. The moment it becomes a document, the machinery above switches on.

We would like that gap to be smaller than it is, and it is on the list. Meanwhile the honest thing is to tell you where the line is rather than let the presence of a checker somewhere in the product imply that everything is behind it.

The claim we are willing to make

We do not publish an accuracy percentage, and we are not going to. Any number of that shape is a claim about every future answer, measured on a sample of past ones, and in a field where the cost of a single bad citation is a lawyer standing up in court to explain themselves, it is the wrong kind of promise to make.

What we will say is what the machinery does. The reasoning agents cannot see reporter citations. A separate desk writes them. Five checks run before a document is released. A paragraph number with no map behind it is deleted rather than guessed. Six hundred and seventy-six citations have been stopped by that process so far, and the log of them is a table we look at, not a marketing asset we tuned.

That is a claim about a process, and a process is a thing you can inspect. It is also the only kind of claim we think anyone in this business has earned the right to make.

---

Counts run against Miss Lucy's own records on 12 August 2026 and will have moved by the time you read this. The 206 untyped entries predate the classification we added in June 2026; they were caught and blocked, we simply did not record which of the five checks caught them.