Building a knowledge base that retrieves well

RAG answers are only as good as the chunks you retrieve. Feed the model the wrong passage and it will confidently answer from the wrong passage. So the real work of RAG is not the prompt, it is making retrieval reliably find the right thing. This lesson is that craft.

Prepare the documents

Clean the source first. Strip navigation, boilerplate, and duplicates. Keep real structure like headings and tables, because that structure carries meaning. Attach metadata to each document (product, category, date, source) so you can filter on it later. Tidy inputs are most of retrieval quality.

Chunking

Split documents into pieces small enough to be specific but large enough to stand on their own. Cut on natural boundaries, a section, a Q&A pair, a clause, rather than every N characters mid-sentence. Add a little overlap between chunks so an idea that straddles a boundary is not lost. Bad chunking is the most common reason RAG underperforms.

Retrieve better

• Vector search finds chunks by meaning; keyword search catches exact terms like SKUs and names.
• Hybrid search runs both and merges, so you get meaning and precision.
• A re-ranker reorders the top candidates by true relevance before they reach the model.
• Metadata filters narrow to the right subset first (this product, this month, this branch).

Ground and cite

Tell the model to answer only from the retrieved chunks, to say plainly when the answer is not in them, and to cite which chunk it used. Citations are not decoration, they let you and the customer verify the answer, and they make wrong answers easy to trace back to a bad chunk.

Measure retrieval

Build a set of real questions, each paired with the chunk that should answer it. Check whether retrieval returns that chunk in the top results. If it does not, no prompt and no bigger model will save the answer, the right information never reached the model. Fix retrieval first, always.