Enterprise RAG Systems

A retrieval-augmented system answers from your documents rather than from what a model absorbed during training. Five things happen between a question and an answer, and each one is a place where quality is won or lost.

Ingestion and chunking. Documents are split into passages small enough to retrieve precisely and large enough to stay coherent. Chunk size is not a universal constant: contracts, wiki pages, support tickets and slide decks each behave differently, and one chunking rule applied across all of them is the most common reason retrieval feels vague.

Embedding and indexing. Each passage becomes a vector and is stored for search. The index needs a refresh strategy from the start — a scheduled rebuild, or change-data-capture on the source. A system that quietly serves last quarter’s policy is worse than no system, because nobody can tell by looking at the answer.

Retrieval. The question is embedded and the nearest passages come back. Pure vector similarity misses exact strings — part numbers, policy codes, proper nouns — so production systems usually combine it with keyword search rather than choosing between them.

Reranking. A second, slower model reorders candidates by actual relevance. It is often the cheapest large improvement available once a baseline exists, which is exactly why it should be added after you can measure whether it helped, not before.

Generation with citations. The model answers from the retrieved passages and points at them. Citations are not decoration: they are what makes a wrong answer detectable by the person reading it, which is the difference between a tool an operations team trusts and one they quietly stop using.

These are the failure modes that show up after launch rather than during the demo.

Four numbers do most of the work, and they are worth agreeing before anything is built.

Retrieval recall asks whether the passage containing the answer came back at all. It is the ceiling on everything downstream: no amount of prompting rescues an answer whose source was never retrieved.

Citation accuracy asks what share of answers cite a source that genuinely supports the claim. An answer with a citation that does not support it is more dangerous than one with no citation, because it borrows credibility it has not earned.

First-pass acceptance asks how often the answer is used without editing. This is the number the business actually feels, and the one worth putting in a business case.

Refusal behaviour asks what happens when the corpus does not cover the question. A system that says it does not know is worth more than one that always produces something, and it is the behaviour most often missing from a pilot. The RAG implementation checklist sets out the checks in order.

In most enterprises the difficulty is not making retrieval work. It is making retrieval respect who is asking. Two approaches are common, and they trade off differently.

Filter at query time. Per-document access rules are stored alongside the vectors and applied inside the retrieval call using the identity of the person asking. This handles fine-grained permissions but means your index has to carry, and keep current, whatever the source systems know about access.

Segment the indexes. A separate index per audience is simpler to reason about and easier to prove to an auditor, but it multiplies indexes and breaks down as soon as permissions vary per document rather than per group.

Either way, the decision belongs before indexing rather than after, and every retrieval should be logged so that someone can later reconstruct who saw what. That log is usually the first thing a compliance reviewer asks for — see the BFSI and healthcare notes for how that plays out in regulated settings.

Discovery takes one to three weeks and settles the corpus, the access rules and the questions the system will be judged on. A working version aimed at a single workflow follows in four to eight weeks. A pilot with real integrations and real access controls runs eight to twelve weeks, with people still in the approval path. Production scope depends on governance and scale rather than on model choice.

If you are still deciding whether retrieval is the right approach at all, RAG versus fine-tuning covers where each one wins. Pricing sets out what each stage costs, and the first two weeks are free.