Naive RAG: What chunk_size=1000 and vector-only search do

This text used to be a how-to. The same stack is below, as a finding, not as a recipe.

In a healthcare project 500+ videos and PDFs sat in Milvus, roughly 50,000 chunks, answers under 100 ms. Legal does not ask for latency. It asks for the passage.

What we built was naive retrieval: chunking by character count, vectors only, no reranking. Hallucinations are then expected.

What has to come after is in Hybrid RAG and reranking.

What the pipeline did

Three stages, as in every tutorial:

Ingestion: load documents, cut chunks, write vectors

Retrieval: similarity search

Generation: top-k chunks into the prompt, model writes the answer

Stack: LangChain, GPT-4o, self-hosted Milvus (Docker), FastAPI, Python 3.12. Self-hosting was right: in that setting the data must not leave the house. The error was not Milvus. It was treating similarity as proof.

Finding 1: fixed chunk size

from langchain.text_splitter import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=1000,
    chunk_overlap=200,
)
chunks = splitter.split_documents(docs)

1000 / 200 was a compromise by feel. Headings, tables, clause numbers: ignored. A chunk split mid-table. The neighbour sounded similar and won the search. The claim in the prompt came from the wrong row.

Chunk quality still beats quantity. Fixed character counts still produce expected errors.

Finding 2: vectors only

from langchain_openai import OpenAIEmbeddings
from langchain_milvus import Milvus

embeddings = OpenAIEmbeddings(model="text-embedding-3-small")
vectorstore = Milvus.from_documents(
    documents=chunks,
    embedding=embeddings,
    connection_args={"host": "localhost", "port": "19530"},
    collection_name="knowledge_base",
)

Cosine similarity finds paraphrases. It does not find file IDs, paragraphs, product codes. Metadata filters helped. We had them. They do not replace keyword search.

Finding 3: unchecked top-k

from langchain.chains import RetrievalQA

qa_chain = RetrievalQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    retriever=vectorstore.as_retriever(search_kwargs={"k": 5}),
)

The five nearest chunks went into the prompt. No reranker. No rule "no source, no claim". "Similar" was treated as "proven". Source grounding requires the opposite.

Evaluation was planned from day one. The set existed. It did not stop the fast pipeline, because the set checked style, not origin.

What I do differently now

Chunks at headings and tables, not at chunk_size=1000

Hybrid: keyword plus vector, then rerank

Hard rule: if the openable source is missing, the system says no

Evaluation against the source, not against how it sounds

The entry is the LLM readiness check. Day 2 is retrieval and grounding. Practice: healthcare chatbot.