How DocuChat catches made-up law references in LLM answers

“Why is DocuChat citing a law that doesn’t exist?”

That question is unpleasant in any AI product. In a legal context, it is especially bad. If the model confidently says that a clause conflicts with a regulatory act, but that act is not in the database, trust in the check ends immediately.

DocuChat checks legal wording in documents. The input can be a short contract excerpt, or it can be a 100+ page document. The LLM is good at reading language, finding problematic passages, and suggesting edits. But it has a common problem: if you ask it to justify a conclusion, it may generate a reference to a “suitable” law that does not exist.

The worst part of this bug was that the answer looked clean. There was no broken formatting or obvious nonsense. The reference was in place, the wording looked legal, and the tone was confident. Manually checking every such output would mean pushing the risk back onto the user.

First, we looked at the prompt and document chunking

The first path was the obvious one: tighten the model instructions. Ask it to reference only known regulatory acts, not to invent numbers, and to explicitly write “no data” if there is no confirmation.

That reduces noise, but it does not eliminate the error class. The LLM is still a text generator. It may not “know” that a specific reference does not exist in the sources, especially if there are similar formulations in the surrounding context.

At the same time, we looked at chunking. For long documents, you cannot just send the whole file to the model: the context window is limited, and both cost and latency increase. So the document is split into chunks, and retrieval then pulls the relevant passages for review.

And that created a second problem. The smaller the chunk, the easier it is to find the exact spot, but the higher the chance of losing the connection to definitions, appendices, or caveats in other sections. The larger the chunk, the better the context is preserved, but retrieval starts bringing back heavier passages where a useful sentence can get buried in the surrounding text.

We were not arguing about “the correct chunk size,” but about where the boundary lies between search precision and legal coherence. For 100+ page documents, this is not an academic question: one caveat at the beginning can change the meaning of a clause in the middle.

The cause was not only context

Losing the “big picture” explained part of the strange answers, but not the made-up regulatory references. If the model did not find an act in the context, it should not substitute a similar one. That meant retrieval alone was not enough.

We split the problem into two tasks:

• analyzing the document text and finding problematic wording;
• confirming every reference to a regulatory act against an authoritative database.

The LLM can suggest what exactly to check and how to phrase the remark. But the law reference goes through a separate layer of ground-truth verification. If a regulatory act appears in the answer, it is checked against the database. No match means the reference is not considered confirmed.

This is an important distinction. The model is no longer the only source of truth for legal grounds. It takes part in the analysis, but the final confirmation of the reference is done by deterministic verification against the database of regulatory acts.

What changed in the pipeline

After generating an answer, we started extracting references to regulatory acts from it and running them through verification. An answer with an unconfirmed reference does not pass as a valid result.

For long documents, we kept the chunking + retrieval setup, but became more careful about balancing chunk size. Smaller chunks help find a specific clause more precisely. Larger ones preserve the context around the clause. The setting depends on the type of document: we did not find a universal size that works equally well for a short agreement and for a 100+ page document.

As a result, DocuChat now has two separate quality loops. The first ensures that the model sees the relevant parts of the document and does not lose the meaning. The second ensures that legal references do not live only in the answer text.

Now the most dangerous class of error is not caught by asking the model to “be more careful,” but by checking against the database. If the regulatory act cannot be found, the system should treat it as an unconfirmed reference, not as a polished justification.