RAG for Internal Documentation: A Small-Team Guide

RAG for internal documentation — retrieval-augmented generation — means finding the few passages from your docs that answer a question, putting only those in the model's prompt, and instructing the model to answer from them and say so when they don't contain the answer. For a small team, the parts that matter most aren't the model: they're documents written to be retrieved (one topic per section, the questions people ask), hybrid keyword-plus-vector search, small prompts, strict grounding, and permission checks so the assistant never retrieves what the asker can't see.

How RAG works, briefly

Retrieval-augmented generation was introduced by Lewis et al. (2020): combine a retriever that finds relevant documents with a generator that writes an answer conditioned on them. The appeal for internal docs is obvious — the model doesn't need to have been trained on your wiki; it reads the relevant bits at question time, and you can update docs without retraining anything.

The pipeline:

1. Ingest — split docs into chunks, store text plus metadata (and embeddings).
2. Retrieve — for a question, find the best chunks.
3. Generate — prompt the model with the question, the chunks and grounding rules.
4. Cite — show which docs the answer came from.

Step 1: write docs that retrieve well

Most RAG quality problems are document problems. Guidelines:

• One topic per section, with a heading that names it. A 40-page "IT handbook" retrieves badly; "How to connect to the VPN from home" retrieves well.
• Put the questions people ask into the doc. LetRelay's in-app guide gives every page a list of real phrasings ("make someone admin", "change someone's role", "promote to admin") alongside the steps. Those phrasings act as retrieval anchors for the ways people actually ask.
• Use the words users use, including synonyms: "admin", "role", "permissions", "position".
• Keep facts current. RAG will faithfully repeat an outdated doc. Every doc needs an owner and a review date.

The same principles make good self-service articles; see ticket deflection with a self-service knowledge base.

Step 2: chunk sensibly

Split along the document's own structure — headings, then paragraphs — rather than fixed character counts that cut sentences in half. Keep each chunk's title and page name with it, so a retrieved chunk makes sense alone. For short internal guides, a whole page is often the right unit; LetRelay's guide retrieves whole feature pages, each a few hundred words.

Step 3: hybrid retrieval

Neither retrieval method is enough alone:

• Keyword (full-text) search is precise on exact terms — product names, error codes, ticket numbers — and weak on paraphrase.
• Vector search over embeddings matches meaning ("can't get on wifi" ↔ "wireless connectivity") and is weaker on exact identifiers.

Combine them and merge the rankings (reciprocal rank fusion is a common, simple method). Add typo tolerance and a synonym list to the keyword side. On LetRelay's guide, a keyword retriever with synonyms and typo tolerance put the right page first on 79% of a 47-question set at baseline; adding embedding similarity raised a separate blind set's top-1 accuracy from 53% to 80%. Implementation details for Postgres are in semantic search with Supabase pgvector.

Step 4: keep the prompt small

More context isn't always better:

• Cost and rate limits. On free tiers, prompt tokens are capacity: Groq's free plan documents 8,000 tokens per minute for openai/gpt-oss-20b. LetRelay cut its how-to prompt from 2,200–2,600 tokens to about 1,900 by sending four guide pages instead of six and a compact feature index.
• Attention. Research ("Lost in the Middle", Liu et al., 2023) found models use information at the beginning and end of long contexts better than information in the middle. Fewer, better chunks — most relevant first — help.

Retrieve maybe 10 candidates, rerank or filter, and send the top 3–5.

Step 5: ground the answer

The system prompt should make grounding explicit:

• Answer only from the provided documents.
• If they don't contain the answer, say so and suggest where to go (file a request, ask a named team).
• Quote exact names of buttons, menus and settings from the documents.
• Never invent links; link only to pages provided.

And scope what the answerer can claim. LetRelay's how-to answerer is told it cannot see workspace data — people, requests, calendars. Before that rule, asked "is Dana available?", it confidently invented an answer. Questions about live data go to a separate, permission-checked path, not to the docs model.

Step 6: permissions

Internal docs aren't all public inside the company: HR policies, security runbooks and finance procedures may be restricted. Retrieval must respect the asker's permissions before anything reaches the prompt:

• Store each chunk with its access scope (organization, team, role).
• Filter by the asker's permissions in the retrieval query itself — with row-level security in Postgres, the database does it.
• Never retrieve broadly and ask the model to "not mention" restricted content; it will.

The permission model is described in role-based access for internal tools.

Step 7: evaluate retrieval and answers separately

Build a golden set of real questions with the doc that answers each. Measure retrieval (right doc in top-1 / top-3) on every change — it's cheap and needs no model calls. Measure answers with must-include and must-not-include checks when prompts or models change. Keep a blind set for honesty. The method is in how to evaluate an LLM feature with a golden set.

Step 8: fall back gracefully

When the model is unavailable or out of quota, RAG degrades well: you already have the retrieved documents. Show the best-matching page's steps and a link instead of a generated answer. LetRelay's assistant does exactly this when every model fails, so a how-to question still gets a useful reply. See AI that degrades gracefully.

Common mistakes

• Chunking by character count — cuts steps in half and loses headings.
• Vector-only retrieval — misses exact product names and codes.
• Huge prompts — slower, costlier, and not more accurate.
• No "I don't know" — the model fills gaps with plausible fiction.
• Retrieval that ignores permissions — the fastest way to leak an HR document.
• No evaluation — you can't tell whether a change helped.

FAQ

What is RAG for internal documentation?

A method where an assistant retrieves relevant passages from your internal docs for each question and writes an answer grounded in them, instead of relying on what the model learned in training.

Do I need a vector database for RAG?

Not necessarily a separate one. Postgres with pgvector plus its built-in full-text search supports hybrid retrieval for a small team's documentation.

How many documents should go into the prompt?

Usually three to five of the most relevant chunks, most relevant first. More context raises cost and can lower accuracy.

How do I stop a RAG assistant from making things up?

Ground it strictly in retrieved documents, allow "I don't know", scope what it can claim, and test with must-include and must-not-include checks.

Sources

• Lewis et al. — Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• Liu et al. — Lost in the Middle: How Language Models Use Long Contexts
• Supabase — Hybrid search
• pgvector on GitHub
• Groq — Rate limits