Retrieval-augmented generation: the technical layer behind an agent that answers correctly

A generalist model like ChatGPT or Claude answers from two sources: what it learned during training, and whatever it is told in the current conversation. Neither of those is your company’s current price list, your leave policy, or last week’s contract amendment — so an assistant left to its own memory will, sooner or later, invent an answer that sounds right and is not. This is not a training-quality problem that a bigger model fixes on its own; it is structural, and the architecture built to address it has a name: retrieval-augmented generation, or RAG.

Search first, generate second

The mechanism is simpler than the acronym suggests. A question is converted into a numerical representation and matched against an index of the company’s own documents; the passages that match best are pulled out and handed to the model along with the question, so the answer is generated from that retrieved material rather than from memory alone. The approach was formalized in a 2020 paper, Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, by a Facebook AI Research team led by Patrick Lewis and presented at NeurIPS — the paper that gave the technique its name and its now-standard two-step architecture.

What changes for a company deploying an assistant

The practical gain is not that the model becomes smarter — it is that its answer is now grounded in a specific, checkable document instead of a statistical average of its training data. Update the source document and the assistant’s next answer reflects it immediately, with no retraining. The catch is that retrieval quality sets a ceiling on answer quality: an assistant that confidently answers from the wrong retrieved passage is still wrong, just wrong with better manners.

• →Chunking: documents split too coarsely or too finely make the right passage hard to retrieve — the split should follow the document’s own logic (one policy clause, one contract article), not a fixed character count.
• →Freshness: a RAG assistant is only as current as its index — anything time-sensitive (pricing, leave policy, this month’s terms) needs a defined reindex schedule, not a one-off import.
• →Traceability: an answer with no visible source is a black box even when it happens to be right — require the agent to show which document it drew from, so a person can verify it in one click before acting on it.

None of this removes the need for human review before a RAG-based assistant goes live on sensitive topics. What it changes is the failure mode: instead of a confidently invented answer with no trace of where it came from, you get a checkable one — wrong sometimes, but wrong in a way you can catch.