RAG in production

A RAG prototype does not need to impress anyone these days. LangChain, an embedding model, a vector database, three days of work, the demo is done. The gap between that demo and a system running in production at a regulated company is large. The demo answers the questions from the test set. The production system has to answer why it made a claim, who can check it, and what happens when the basis is missing.

This piece is the overview of the path from demo to that system. I have described each station elsewhere in detail, with code and with findings from real projects. Here they sit in context, in the order I check them in a review.

Naive RAG is not production

Naive RAG looks the same in every tutorial: load documents, cut them into chunks, write vectors, search the nearest chunks for a question and hand them to the prompt. I built exactly that, in a healthcare project, with LangChain, Milvus and GPT-4o. 500+ videos and PDFs, roughly 50,000 chunks, answers under 100 ms. The pipeline was fast. It was not automatically correct.

Naive retrieval has three traits that never show up in a demo and generate a ticket every week in production:

chunking by fixed character count, ignoring headings, tables or paragraphs

vector search only, no keyword matching for file numbers, clause references or product codes

top-k chunks go into the prompt unchecked, "similar" gets treated as "proven"

The gap between demo and production is not scale. A system that answers wrong for ten users answers just as wrong for a thousand, only more often. The gap is whether someone can explain the answer afterward. In the demo nobody asks. In production, legal asks, the customer asks, or an auditor asks.

Chunking: the first source of error

chunk_size=1000, chunk_overlap=200: this two-liner sits in every second RAG tutorial, and it sat in at least one of my own projects too. It ignores the structure of the document completely. A table tears apart mid-row. A heading lands in the last third of one chunk, the rest of the section in the next. In search, the chunk that happens to contain more matching words wins, not the chunk that carries the answer.

What works instead:

chunking along document structure: headings, paragraphs, table rows stay together

smaller chunks for dense technical text, larger ones for prose

metadata per chunk: document, version, section, page

Structure-based chunking, in practice, means the splitter knows the outline, not just the character count.

from langchain_text_splitters import MarkdownHeaderTextSplitter

headers = [("##", "section"), ("###", "subsection")]
splitter = MarkdownHeaderTextSplitter(headers_to_split_on=headers)
sections = splitter.split_text(document_markdown)

Each section now carries its heading as metadata, not just the running text. For PDFs without markdown structure, you need layout detection first, something that tells headings, tables and footnotes apart before any splitting happens. That extra step costs time during ingestion. It saves time later, when a reviewer wants to know why a chunk got cited.

That last point above gets skipped often. Without metadata, nobody can later say which version of a document an answer came from. That is context management, not just chunking. Upload a new version of a manual and leave the old one in place, and sooner or later an answer comes from the outdated version. Chunk quality beats chunk quantity, but only if provenance is attached to the chunk.

Hybrid retrieval instead of vector search alone

Vector search finds paraphrases well. It does not reliably find a clause number, a file reference or a product code, because embeddings are trained on meaning, not exact strings. A question about "§ 14 para. 2" can sit semantically close to a completely different paragraph and still come back as the top hit.

Hybrid search combines keyword search, BM25 or a full-text index, with vector search and merges both result lists:

def hybrid_search(query: str, vectorstore, bm25_index, k: int = 20):
    vector_hits = vectorstore.similarity_search(query, k=k)
    keyword_hits = bm25_index.search(query, k=k)

    seen = {}
    for hit in vector_hits + keyword_hits:
        seen[hit.id] = hit
    return list(seen.values())

This is not finished retrieval, just the first step: both channels deliver, then results get merged and deduplicated. Vectors only and you lose exact matches. Keywords only and you lose paraphrases and synonyms.

Simple merging like above is fine for a first pass. Reciprocal rank fusion works better: instead of just deduplicating hits, RRF weights each hit by its rank in both lists, not by the raw score, because BM25 scores and cosine similarity sit on completely different scales and do not add up in any meaningful way.

def rrf_fuse(vector_ranked: list[str], keyword_ranked: list[str], k: int = 60):
    scores: dict[str, float] = {}
    for rank, doc_id in enumerate(vector_ranked):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    for rank, doc_id in enumerate(keyword_ranked):
        scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)
    return sorted(scores, key=scores.get, reverse=True)

RRF does not replace reranking afterward. It is the better input for it: a ranking that weighs both search paths fairly before the reranker sets the final order by the question. Both together, without these two ranking steps, still just gives a longer list in roughly the right order. That is not enough.

Reranking: order by the question, not by similarity

A reranker takes the combined hit list and sorts it again, by the actual question, not by cosine similarity or BM25 score. Models like Cohere Rerank or a self-hosted cross-encoder score each pair of question and chunk individually:

def rerank(query: str, candidates: list[Chunk], reranker, top_n: int = 5):
    scored = reranker.score(query, [c.text for c in candidates])
    ranked = sorted(zip(candidates, scored), key=lambda x: x[1], reverse=True)
    return [chunk for chunk, score in ranked[:top_n]]

Without this step, the loudest neighbour often wins: a chunk full of matching terms that does not answer the question. With reranking, the number of wrong top hits drops noticeably. That is why hybrid RAG and reranking get named together instead of treated as two separate options. Hybrid without reranking is a better hit list. Hybrid with reranking is retrieval that can carry an answer.

Source grounding: no claim without proof

Reranking improves the hit list. It does not stop a model from inventing something when the hit list is empty or weak. That is where source grounding starts: a hard rule in the flow, not a polite instruction in the prompt.

Three requirements I enforce in every project:

Every substantive claim is attached to a chunk a human can open.

If no matching chunk exists, the system says so. It does not fill the gap with a plausible guess.

The interface shows the source next to the answer, not in a separate log nobody opens.

"Be helpful, but don't make things up" in the prompt is not enough. A model follows that while the question is simple, and drifts as soon as the hit list gets thin. The rule has to sit in the flow: retrieve, check whether enough evidence exists, only then generate. Skip that step and the system eventually produces a paragraph that reads like it came from a standard and did not. Details in LLM hallucinations are a compliance problem.

Evaluation: measure instead of hope

Every change to the system, a new model, a new prompt, a new index, can shift quality in either direction. Without evaluation, nobody notices until a user complains or an auditor asks.

What an evaluation set needs:

a fixed set of questions with an expected source or expected decision

someone who maintains the set as documents change

a run after every model, prompt or index change, not only before launch

Tools like RAGAS calculate metrics such as context precision or faithfulness automatically. They do not replace the decision of what to measure. A typical finding: a team switches from a smaller to a larger model because the text reads more fluently. The source-grounding rate drops at the same time, because the larger model states things more confidently, even without evidence. Without a set that measures that rate, nobody sees it before the next audit. More in LLM evaluation in production.

Observability: a bad answer has to be reconstructable

A complaint comes in: wrong answer, three days ago, details forgotten. Without observability, that case is not workable. What has to be logged per request so a run can be rebuilt later:

which prompt, which version

which chunks, in which order, including the reranking score

which model, how many tokens, what cost

what the user actually saw

A dashboard with token totals is FinOps, useful, but a different question. It does not answer why a particular sentence invented a particular clause reference. Tools like Langfuse map that path, question to chunks to answer, in one place a reviewer can open without the user's chat history. The name of the tool is secondary. The path is the requirement. More detail in a bad answer has to be reconstructable.

Tenant isolation: a pilot is not a product

A pilot runs with one customer, one index, one prompt. The second customer arrives, and suddenly two files sit in the same vector space. Without tenant isolation, the system stays a one-customer tool, no matter how many contracts get signed.

The entire retrieval path has to be isolated, not just the interface:

a filter or index per tenant, enforced in code, not in query intent

no chunks crossing the tenant boundary, not even as "similar" context

secrets and keys not sitting in a shared repo default

agent state that does not carry over when the tenant changes

The most common finding is a shared vector space with a metadata filter that goes missing once during a reindex. The answer then cites the neighbour, and nobody notices, because the answer still sounds plausible. Full detail with code in tenant isolation for RAG and agents.

Agents, LangGraph and MCP: the same rule still applies

Once RAG becomes an agent that calls tools, the task changes, the rule does not. LangGraph orchestrates steps, retrieval, tool call, check, answer, as a graph instead of a single chain. That makes intermediate steps visible and enforceable: one node can check whether enough evidence exists before the next node even starts.

MCP servers bring external tools into that flow, a ticketing system, an internal API, a search engine for product data. Every tool call is one more point where a claim can appear that nobody can verify. An agent that calls an MCP tool and writes the result into the answer unchecked has the same problem as a RAG system without source grounding, just with one more source of error. The source-grounding rule applies to every node in the graph, not only to the retrieval step at the start. Build an agent without applying that rule to every tool call, and the result is a retrieval problem with more moving parts.

Operations: latency, cost, scale

In the healthcare project with 500+ videos and PDFs, roughly 50,000 chunks ended up in Milvus, and answers came back under 100 ms. That was operations, not just architecture. An operating model means someone decided in advance how many chunks per second the vector database has to handle, how often reindexing runs when documents change, and how much extra latency a reranking step is allowed to cost.

Reranking costs time. A cross-encoder over twenty candidates is slower than a plain vector search over five. That is an acceptable trade-off when the added latency sits in the low double-digit milliseconds and hit quality improves noticeably. It is not an acceptable trade-off when nobody set the budget beforehand and a user gives up after three seconds.

Cost drifts unnoticed in a similar way. Every extra chunk in the prompt costs tokens, on every single request, not once. A reranking step that trims the candidate list from twenty to five before the context goes into the prompt lowers that cost noticeably while improving quality at the same time. Anyone not measuring this notices the increase only on the invoice at the end of the month. Practice with numbers from a running system: the healthcare chatbot.

The path from demo to operations

None of these points is hard on its own. Hybrid search is one extra search path. Reranking is one extra call. Source grounding is one rule in code. Together they are the building blocks between a demo and a system an auditor lets through. The order I check this in during a review, correct chunks, hybrid retrieval, reranking, source grounding, evaluation, observability, tenants, operations, is exactly the order of this piece.

Anyone who wants to know where an existing system sits on this list gets that from the LLM readiness check: five days, a written finding, starting at €5,000, no obligation to buy the follow-up project. Implementation itself sits under AI engineering.