AI and RAG · 7 min
RAG for business: when document search is worth it
A deep guide to AI search over internal data: applicability, sources, permissions, answer quality, cost, security and the path from prototype to production.
First decision
Not every text problem needs RAG
Start with the question type: an exact filter, a rule or an answer that depends on many documents.
Answer path
The answer starts before the model
A reliable system connects sources, permissions, retrieval and evaluation.
RAG is not a chat window over files
RAG helps when an answer cannot be obtained from one database field. An employee may need to combine a contract, policy, instruction and current status, then understand the evidence behind the result.
A demo can load files and produce convincing text. Production needs freshness, access control, repeatability, safe refusal and a way to investigate errors.
When traditional search is better
If a user knows an order number, date, status or exact attribute, a structured query is faster and more predictable. AI should not replace filters just because a chat feels modern.
The best product often combines both: structured data through exact queries, RAG for explanations and evidence across documents.
Data and permissions come before model choice
Map owners, freshness, update rules and access for every source. An outdated document can produce a harmful answer even with excellent retrieval.
Apply permissions before context is assembled. Finding a forbidden fragment and hoping the model will hide it is not a security boundary.
Measure quality on a question set
Build 30–100 representative questions with expected answers and acceptable sources. Check relevance, completeness, citations, latency and safe refusal.
This turns the question set into a regression test for indexing, prompts and models instead of relying on a few impressive conversations.
Cost includes operations
Budget includes ingestion, re-indexing, versioning, access control, logging, evaluation and support—not only model tokens.
Start with one department and a limited source set. Prove value and operating cost before widening the knowledge boundary.
From prototype to production
First prove one measurable journey. Then add permissions, observability, evaluation, failure handling, source refresh and human escalation.
A useful AI product can say “I do not have evidence”, show why and hand the question to a person. The engineering value is in the controlled chain.
Start with a question that needs more than one source
RAG is useful when an answer must combine documents, respect user permissions and show evidence. If a value lives in one table and follows a strict rule, a normal API will be faster, cheaper and more reliable.
Collect real questions before choosing a model. Include short and long queries, alternate wording, outdated documents, conflicting policies and questions with no answer. This set becomes a regression test for quality.
Quality starts with source preparation
Uploading files is not knowledge preparation. Define ownership, validity dates, versions, language, access level and links to related material. Preserve headings, tables and source-page references during chunking.
Indexing must be repeatable: the same document should not create duplicates, and a changed policy should remove its old version. Check permissions before retrieving fragments, not after generation.
Production RAG needs a safe refusal
An answer without evidence is a quality failure, not a success. Show citations, document dates and confidence signals, and say when the system cannot verify a conclusion.
Track grounded-answer rate, retrieval latency, request cost, reformulations and unanswered questions. This evidence improves chunking and retrieval before a model change is considered.
Find the source of truth first
RAG does not automatically repair chaotic documents. Identify owners, publishing rules, freshness windows and versions before choosing a model. The same term can have different priority in a contract, a manual and a spreadsheet.
Create a small benchmark of questions with expected answers and citations. It shows whether the problem needs multi-source retrieval or whether a form, filter and conventional full-text search are enough.
Answer quality has several checks
Users judge more than generated prose. Retrieved fragments, permission checks, useful citations and an honest insufficient-context response all matter. Test retrieval, generation and failure behaviour separately.
Include precise facts, document comparisons, missing answers and queries for restricted data. This reveals system boundaries much faster than a happy-path demo.
Operations matter more than a demo chat
Production systems have changing documents, users and access policies. You need re-indexing, removal of revoked sources, query audit and an explanation of why a fragment was used.
Start with a bounded corpus and measure cited-answer rate, retrieval latency and request cost. Only then decide where a larger model or an agent workflow adds value.
Measure the cost of a wrong answer first
RAG is not justified by adding a chat box. Ask what a mistake causes: an hour lost, a wrong customer commitment or a compliance issue. The higher the cost and the more sources must be reconciled, the stronger the case for controlled retrieval.
If the answer always lives in one table, strict search and forms may be cheaper and more reliable. Architecture should follow risk, not a fashionable label.
Source preparation shapes quality more than the model
Before indexing, capture ownership, version, validity and access level. Remove duplicates, add headings and preserve links to source fragments. Otherwise retrieval may find similar text without proving that it is current.
Build a pilot set of real questions and expected sources. Evaluate citation completeness, permission correctness and refusal quality, not only semantic similarity.