RAG chatbots: retrieval-augmented generation for answers grounded in your documents

The loop

1. Ingest: collect documents, clean them and split them into chunks.
2. Index: compute an embedding for each chunk and store it with metadata (source URL, title, date, access group).
3. Retrieve: for each question, find the most relevant chunks.
4. Generate: give the model the question plus those chunks and ask it to answer only from them, with citations.

Chunking

• Split on structure first (headings, paragraphs), then by size. A few hundred tokens per chunk with a small overlap is a common starting point; tune it with evaluation, not intuition.
• Keep the heading path with each chunk ("Billing > Refunds > Annual plans") so it makes sense on its own.
• Store the original text and a stable ID so you can update or delete chunks when the source changes.

Retrieval

• Semantic search with embeddings finds paraphrases. Keyword search (BM25) finds exact codes, names and error messages. Hybrid search combining both is usually more robust than either alone.
• A reranker that scores the top 20 to 50 candidates against the question often improves precision a lot.
• Filter by metadata before ranking: language, product version and, critically, what the asking user is allowed to see.

Prompting with context

Answer using only the sources below. Cite them as [1], [2].
If the sources do not contain the answer, say you do not know and suggest contacting support.

<sources>
[1] (Refund policy, updated 2026-03-01) ...
[2] (Plans FAQ) ...
</sources>

Wrap retrieved text in delimiters and tell the model it is data. Documents can contain instructions written by someone else; see prompt injection defense. Build and test prompts with the prompt template builder and check sizes with the token estimator.

Permissions and freshness

• Never index content into a shared space and rely on the prompt to hide it. Enforce access in the retrieval query.
• Re-ingest on change (webhooks from your CMS or a scheduled job) and show the source date in answers.
• Remove deleted documents from the index promptly, especially personal data.

Evaluation

Create a set of real questions with the expected source documents and reference answers. Track:

• Retrieval recall: is the right chunk in the top k?
• Groundedness: is every claim supported by the cited source?
• Answer quality and refusals: does it say "I do not know" when it should?

Re-run the set whenever you change chunking, the embedding model, the prompt or the LLM. See testing bots.

Common failures

• Answers from the model's memory instead of the sources: tighten the instruction and lower temperature.
• Right document, wrong chunk: improve chunk boundaries or add a reranker.
• Out-of-date answers: the index is stale or old versions were not removed.