Retrieval-augmented generation

Retrieval-augmented generation (RAG) is a technique where an AI workflow retrieves relevant documents from a knowledge base and includes them in the model's context before generating a response. It's how most enterprise workflows ground answers in proprietary content.

Reviewed by Transformation.ai editorial · Markdown version · Stage: Pilot

What RAG is

RAG has two halves. The retrieval half pulls the most relevant documents from a searchable corpus — usually via vector similarity, keyword matching, or a hybrid. The generation half takes those documents plus the user's question and produces a response grounded in the retrieved content. Together they let a workflow answer questions about specific proprietary content, instead of hallucinating from the model's training data.

The "augmented" part is the key. The model isn't being retrained on your data. It's being given your data as retrieved context, per query. That distinction matters: RAG doesn't require a training pipeline, doesn't embed your data into weights, and updates instantly when the source corpus updates.

What RAG actually solves

RAG solves one specific problem well: grounding answers in content the model was not trained on. Enterprise workflows need to answer questions about policies, products, past tickets, contracts, and internal documentation. Without RAG, every such answer comes from the model's general knowledge — which is either wrong (it doesn't know your specific policy) or retroactively invented.

RAG also gives you two secondary benefits: citation — the response can name the source it drew from, which is auditable — and staleness handling — when the source corpus updates, the next query retrieves the new content. There's no retraining lag.

What RAG does not solve is the entire hallucination problem. See the next section.

Silent retrieval failure

The failure mode most pilots miss: the workflow "works" — it returns a confident, fluent, well-structured answer — but it retrieved the wrong document, or an outdated one, or the right one with the wrong passage, and generated an answer grounded in content that doesn't apply to the question. The output looks like success. It isn't.

The reason this is dangerous is that it doesn't look like the model hallucinating a citation — the citation is real, the document is real, and the quote exists in the document. Only the interpretation is wrong.

Defenses against silent retrieval failure:

• Name the retrieval source as a mid-stage data input. In the workflow map, the retrieval is not one transform — it's a data input, with quality and staleness captured explicitly.
• Force citations to the passage, not the document. A citation to a 20-page policy doc doesn't help; a citation to the specific section does — and it makes it much easier for the reviewer to spot when the wrong section was retrieved.
• Test retrieval separately from generation. Unit evaluators should catch bad retrieval distinctly, not just bad output.
• Add HITL review where the interpretation matters. Automated review can catch literal errors; humans catch interpretive ones.

What RAG is not

• Not a fix for hallucination. RAG converts some hallucination into a retrieval problem — a different failure mode, not an eliminated one. See hallucination.
• Not a model. RAG is a technique — the pattern of retrieval plus generation. The model is inside it; RAG is a system, not a component.
• Not a substitute for clean data. A RAG system running over a stale knowledge base produces confident, wrong answers with a valid citation attached. Garbage in, cited garbage out.
• Not a one-time setup. Retrieval quality drifts as the source corpus updates, as queries shift, and as the workflow accumulates patches. The retrieval pipeline is a living artifact, like the map itself.