Grounding: how to stop an insurance AI making things up
Retrieval is not enough on its own. A four-stage pipeline — intent, retrieve, generate, validate — is what makes an answer defensible to a regulator.
Masoud Alhelou
Heard
In most industries, a confidently wrong AI answer is embarrassing. In insurance, it is a mis-statement of contractual cover — and depending on where you operate, potentially a regulatory breach. The bar for "usually right" is not met by anything a general-purpose assistant does out of the box.
Retrieval alone is not grounding
The standard approach is retrieval-augmented generation: find relevant passages, put them in the prompt, ask the model to answer using them. This helps enormously. It also fails in a specific and dangerous way — the model blends retrieved text with its own prior knowledge, and the result reads exactly as authoritative whether or not the retrieved documents actually supported it.
Insurance documents make this worse. Policy wordings are full of near-identical clauses that differ in one exception. Retrieve the wrong one and the answer is fluent, specific, well-cited and wrong.
Four stages, not one call
What works is refusing to treat this as a single model call. Understand the intent first — is this a cover question, a claims question, a servicing request? Retrieve against that intent, scoped to the documents that apply to this specific customer's schedule. Generate the answer. Then, separately, validate that answer against what was actually retrieved.
The validation pass is the part people skip, and it is the part that earns the deployment. It asks a narrow question with a checkable answer: is every claim in this response supported by the retrieved text? When the answer is no, the agent does not send it. It says it is not certain and escalates.
Citations are not decoration
Every answer should name the document and section it came from — visible to the customer, and permanently attached to the transcript. This is not a trust-building flourish. It is the mechanism by which a compliance team can audit six months of conversations without taking anybody's word for anything.
It also has a useful second-order effect: when the citation is wrong, it is obvious. An unsourced wrong answer hides. A wrong answer citing the wrong clause announces itself to the first person who reads it.