How to implement a RAG system in a company: data, access, quality and the first release

Corporate AI is useful when it answers a specific question faster and more reliably than the current workflow. Start with one scenario: find a policy, explain a request status or combine several sources.

If the goal is simply “add AI”, quality and scope cannot be defined.

Documents have versions, owners, permissions, tables and attachments. Decide which sources are current, how stale fragments are removed and how poor OCR is handled.

Answer quality is bounded by context quality. More documents do not automatically create more value.

An assistant must not see a document just because it was indexed. Enforce permissions while retrieving fragments and keep the decision auditable.

Citations, refusal when context is insufficient and a separate admin path matter for sensitive data.

Create a small reference set with expected sources, unacceptable answers and completeness criteria. Measure citation accuracy, refusal behavior and latency stability, not only text similarity.

This shows whether to improve chunking, retrieval, prompts or the model.

Start in assist mode: show the answer and sources and let users report errors. Automate actions only after quality and ownership are proven.

Production also needs limits, privacy-safe request logs, cost monitoring and a clear way to disable the scenario.

Choose one question type and one audience: employee policy, contract search or operator support. A limited contour is easier to evaluate and secure than a universal chat over every file.

Define forbidden answers, mandatory sources and conflict handling. These rules belong in acceptance criteria, not only in a prompt.

Filter access while selecting fragments. Do not build a shared context and hope the model will avoid a restricted document.

Keep source, owner and access scope attached to every fragment. Role and document changes should update the index predictably and leave an audit trail.

Create reference questions with expected sources and acceptable answers. Test retrieval recall, citation accuracy, refusal and alternate wording.

Add real failures to the evaluation set after launch. This makes improvement measurable rather than dependent on a few good conversations.

Define which documents enter the index, who can see them and how quickly a revoked source disappears. Preserve version, validity date and origin with every chunk.

When structure changes, re-index a controlled area instead of silently mixing old and new fragments. This improves explainability and auditability.

Do not ask a model to avoid secrets and consider the problem solved. Enforce permissions before generation and cite only authorised fragments.

For sensitive data, validate the source document and log the decision. Otherwise a polished paraphrase can still become a leakage channel.

Keep benchmark questions and expected sources and run them after chunking, model or index changes. Track recall, precision, latency, cost and safe refusal rate.

User feedback is actionable when “not found”, “wrong source” and “unsupported answer” are separated. Each points to a different engineering fix.

The system must say when sources are insufficient and explain what is missing. A confident unsupported answer is worse than an empty result for policies, finance and operations.

Include out-of-domain questions, conflicting versions and inaccessible documents in evaluation. Happy-path tests overstate readiness.

Indexing, retrieval, reranking and generation have different load profiles. Set a per-query budget, cache policy and response target for long documents.

Observability should link the question, retrieved fragments, prompt version and answer without exposing sensitive content. This makes quality improvements evidence-based.

Policies, contracts and instructions have different priority and validity windows. Store date, owner and scope as metadata and make retrieval rules explicit. The model should not guess which revision wins.

Include short answers, multi-document comparisons, missing data and out-of-domain questions. Mark expected citations and acceptable “I do not know” outcomes. This makes changes comparable instead of optimizing one lucky example.

Define who uploads a revision, retires the old one and confirms index freshness. Run spot checks for key questions after updates. Otherwise RAG slowly answers from an obsolete policy even when the model is unchanged.