LLM hallucinations are a compliance problem

The demo works. The model sounds sure. Then legal asks: where did sentence three come from? Nobody can show it.

That is where AI projects die in regulated shops. Not at the model. At missing source grounding.

What hallucination means here

The model delivers a paragraph that reads like the standard. The clause does not exist. Or it exists and says something else. In a chat about recipes that is annoying. In an audit finding or patient information it is a compliance case.

I see this regularly. Teams tune the prompt. The rate drops a bit. Legal's question stays the same: can you prove the claim?

Source grounding, not gut feel

RAG alone is not enough. Vector search fetches similar chunks. Similar is not proven.

What I need in projects:

Every domain claim hangs on a source a human can open.

If the source is missing, the system says so. It does not invent a paragraph.

The UI shows the evidence, not only the prose.

Sounds simple. In practice point 2 is often missing. The model fills gaps because the prompt says "be helpful". Helpful without a source is the opposite in an audit.

Why prompt tuning does not solve it

You can forbid the model to guess. It still guesses, only more politely. Without a retrieval layer with a hard rule (no source, no claim) this stays a wish.

Evaluation makes the rest visible. A prompt or model change can double the hallucination rate. If nobody measures, the auditor notices first.

What I check in such a review

Is it a citation or only "inspired"?

Do the chunks come from the right document, the right version?

Is there a no when the base is missing?

Can a reviewer follow the path without reconstructing the chat?

That is LLM traceability. LangGraph or LangChain are tools. The rule sits in the flow: retrieve, check, then write.

Who this is for

If you want a demo, you do not need this. If you want to go live and legal sits in the building, you need it before rollout. That is where the buy happens.

If you are at that point: the LLM readiness check is five days, in writing, with no obligation to continue.