Hybrid RAG and reranking instead of naive vector search

Chunking by character count, vectors only, no reranking: naive retrieval. Hallucinations are then expected.

The older piece RAG pipeline with LangChain and Milvus describes exactly that path from a healthcare project. Useful as a diagnosis. Dangerous as a recipe. This page is what has to come after.

What naive means

Fixed chunk_size and overlap, ignoring headings or tables

Cosine similarity only

Top-k chunks go into the prompt unchecked

"Similar" is treated as "proven"

In that project 500+ videos and PDFs sat in Milvus, roughly 50,000 chunks, answers under 100 ms. Fast. Not the same as correct. Legal asks for the passage, not the latency.

What hybrid changes

Keyword search finds clause numbers, file IDs, product codes. Vectors find paraphrases. Together, then a reranker that orders by the question. Only then may a chunk enter the prompt.

Without a reranker the loudest neighbour often wins, not the passage.

What source grounding still requires

Hybrid RAG is still retrieval. Source grounding requires: no claim without an openable source. The reranker lowers the error rate. The rule stops the gap.

What the check does with it

Day 2 of the LLM readiness check is retrieval and grounding. I look for vector-only setups and for a chunk a human can open.

Related: evaluation, tenant isolation. Service: AI engineering. Practice: healthcare chatbot.