The failure mode · Pilot stage

Hallucination

A hallucination is when an AI workflow generates output that is fluent but factually wrong — a fabricated citation, an invented number, a plausible-sounding falsehood. In enterprise workflows, most "hallucinations" are actually retrieval failures.

Reviewed by Transformation.ai editorial · Markdown version · Stage: Pilot

What hallucination is

A hallucination is dangerous precisely because it doesn't look like an error. It looks like output. Citation formats are valid, the prose is confident, the sentence structure is grammatical. A human reading the output cannot tell — from the output alone — whether it's correct. They must check against an external source.

In consumer chat interfaces, this is annoying. In enterprise workflows — customer support replies, policy questions, financial summaries, legal research — it's a category of brand and compliance risk. A hallucinated number in a board deck is different from a hallucinated number in a Slack channel.

The four causes — and why diagnosis matters

Hallucination is not one failure. It's four distinct failures with four distinct fixes. Applying the wrong fix wastes time and doesn't reduce the rate you're actually observing.

Cause	What it looks like	Fix
Bad retrieval	Answer is fluent and cites a real document — but the wrong one. The quote exists; the interpretation is off.	Fix the retrieval pipeline: better chunking, better ranking, better source data. See RAG.
Prompt ambiguity	The model interprets the question in an unexpected way and answers that question, non-fabricated but not what was asked.	Tighten the prompt — constrain the output format, add explicit refusal instructions.
Model overconfidence	The model doesn't know and generates a plausible answer instead of saying so.	Add refusal paths and confidence signalling. Use evaluation to measure calibration, not just accuracy.
Distribution shift	The workflow was accurate on cases that matched its tuning data, but new query types aren't in the tuning set.	Refresh the eval set against the current distribution. See model evaluation.

Diagnose before redesigning. "Let's switch models" is the most common response to hallucination, and it fixes one of the four causes — overconfidence — while leaving the other three unchanged.

Why HITL catches it (when HITL is real)

Hallucination is the failure class HITL exists to catch. But HITL only catches it if the reviewer can:

See the source. A citation to the specific passage, not just the document, so the reviewer can spot-check without re-reading twenty pages.
Know the domain. A reviewer who can't evaluate the claim can't catch a hallucinated claim. Assign reviewers by subject-matter relevance, not queue order.
Have a real override. An override that doesn't reach the workflow — the deletion, the correction, the flag — is documentation, not review.

HITL where reviewers can't see sources, don't know the domain, or can't override is theater. See the HITL entry for the three requirements.

What hallucination is not

Not the same as error. All AI outputs can be wrong. Hallucination specifically is confident fluency masking wrongness — the output looks right.
Not unique to LLMs. Older ML systems also produce confident wrong output. The generative case is more visible because the output is fluent prose.
Not fixable in one way. The four causes need four fixes. There is no "hallucination reduction" button.
Not a reason not to ship. A workflow with the right HITL and eval regime can run at a hallucination rate well below what human reviewers miss in manual processes. The question is whether the rate is measured, not whether it's zero.

The diagnosis rule Before you switch models, measure the retrieval quality. In enterprise workflows, most "hallucinations" turn out to be retrieval failures — the workflow cited a real document, just the wrong one. Fix the map before you fix the model.

Frequently asked questions

What is hallucination in AI?

Hallucination is when an AI workflow generates output that is fluent and confident but factually wrong — a fabricated citation, an invented statistic, a plausible-sounding falsehood. It's dangerous because the output is indistinguishable from correct output without external checking.

How do you reduce hallucination?

Diagnose the cause first. Four distinct causes require four distinct fixes: bad retrieval (fix the retrieval pipeline), prompt ambiguity (tighten the prompt), model overconfidence (add citations and refusal paths), and distribution shift (refresh the eval set). Applying the wrong fix wastes time and doesn't reduce the rate of hallucination you're actually seeing.

Is hallucination a model problem or a data problem?

Usually both, but in enterprise workflows it's more often a data problem in disguise. A workflow with RAG that retrieves the wrong document will hallucinate confidently — with a real citation attached. Diagnose before redesigning the model choice; in most cases the fix is in the retrieval source's quality, freshness, or coverage, not the model.

Where this fits

Hallucination is a Pilot-stage design problem

Pilot is where you map the retrieval source, design the review boundary, and set the eval set that measures the hallucination rate. Fixing it in the workflow is cheaper than fixing it in an incident.

See the Pilot stage →