RAG Is Not Search with Better Marketing
RAG demos are easy. Upload a PDF, ask a question, watch the model answer in a confident tone.
Production RAG is harder because the answer is the final link in a chain. Ingestion may be stale. Chunking may have split the decisive sentence from its condition. Retrieval may favour semantically similar nonsense. Permissions may be wrong. The model may cite a document that does not support the claim.
RAG is not search with better marketing. It is an evidence system.
Retrieval has several instruments
Azure AI Search supports keyword, vector, semantic and hybrid retrieval. Each sees relevance differently.
Keyword search is precise when users and documents share important terms. Vector search finds conceptual similarity when language differs. Hybrid search combines textual and vector results. Semantic ranker reranks an initial result set using language understanding.
There is no universal winner. Product names, policy numbers and error codes often need lexical precision. Natural questions and paraphrases benefit from vectors. Hybrid retrieval with semantic ranking is a strong starting point, but only an evaluation set can tell you whether it is right for your corpus.
Chunking is a business decision
Fixed token windows are convenient, not intelligent. A legal clause, table row, troubleshooting step and product description have different boundaries.
Preserve headings, document identity, effective dates, access metadata and stable source URLs. Store enough surrounding structure for a retrieved fragment to remain interpretable. If a paragraph says “this is prohibited” but the chunk loses what “this” refers to, the vector is accurate and the answer is useless.
Permission belongs before generation
Security trimming must happen during retrieval. Do not retrieve forbidden content and ask the model not to mention it. Filter by the caller's authorised scope, and test adversarial combinations of users, groups and documents.
The model cannot leak evidence it never receives.
Citations need a contract
Store fields that support citation quality: title, version, URL, page or section and document owner. The response should distinguish claims supported by retrieved evidence from general explanation. If evidence is insufficient, the correct answer is not a fluent guess. It is a narrow answer or a request for better evidence.
I prefer citations close to the claim, with a direct path to the source. A bibliography at the bottom is not enough when the reader cannot tell which sentence it supports.
Evaluate the retrieval before the prose
Create a test set from real user questions, including difficult negatives and permission boundaries. Measure retrieval with metrics such as MRR and NDCG where appropriate, but inspect failures manually. Then evaluate groundedness, relevance, citation correctness, safety, latency and cost at the answer layer.
When an answer is wrong, identify the failing stage:
- Was the source content current and correctly parsed?
- Did the correct chunk exist?
- Did retrieval return it?
- Did ranking place it high enough?
- Did the prompt present it clearly?
- Did the model use it correctly?
Changing the model is only one possible fix.
The production standard
A good RAG system does not merely sound informed. It can show what it knew, why that evidence was available to this user, how it ranked the evidence and whether the final claim stayed inside it.
That is less magical than the demo. It is also far more valuable.
Ingestion is a product pipeline
The answer cannot be fresher than the index. Design ingestion with the same seriousness as an operational data pipeline.
Record source identity, version, owner, classification, effective date, expiry date, checksum and ingestion status. Make updates idempotent. Detect deletions. Quarantine files that fail parsing rather than silently indexing half a document. Preserve tables, headings and page relationships when they carry meaning.
Decide the freshness objective per source. A policy library may tolerate an hourly sync. Incident data may need minutes. A contract repository may require event-driven updates and legal-hold awareness. “We re-index nightly” is not a design until the business consequence of twelve-hour-old evidence is understood.
Expose index health to the application. When a critical source is stale, the assistant should narrow its answer or warn the user instead of speaking with yesterday’s confidence.
Query understanding is a separate stage
Users do not write search queries. They ask questions with pronouns, history and implied scope.
The retrieval layer may need to resolve entities, expand acronyms, identify time range, infer document type and generate lexical and semantic variants. An agentic system can decompose a broad question into several retrieval steps, inspect evidence and search again.
But query rewriting can also destroy precision. “Show error TS-999” should retain the exact identifier. “What did the latest policy change?” needs a date-aware filter, not only semantic similarity. Preserve the original query and log each transformation so failures can be diagnosed.
For sensitive applications, query understanding must not expand authority. A rewritten request still retrieves only within the caller’s allowed scope.
Reranking buys focus
Initial retrieval favours recall: gather a candidate set large enough to contain the answer. Reranking favours precision: decide which candidates deserve scarce context.
This separation is useful because embedding similarity is not the same as answer relevance. A reranker can compare the user’s full question with each candidate and move the decisive clause above several thematically similar passages.
Tune candidate count, reranked count and final context size together. Too few candidates miss evidence. Too many increase latency and reranking cost. Too much final context distracts the model and can introduce contradictory versions.
Always carry provenance through the reranker. A high score should not erase title, section, date, permissions or source URL.
Context assembly is editorial work
Retrieved chunks do not become good context merely by being concatenated. Group related evidence, remove duplicates, preserve source boundaries and put the most decisive material where the model can use it. Include the user’s question, relevant conversation state and response contract without burying evidence under instructions.
When sources conflict, do not quietly select one. Prefer current and authoritative material according to explicit policy, and make disagreement visible when it matters. A policy approved yesterday should outrank an obsolete wiki page even if the wiki is semantically closer.
Context has a budget. Spend it on evidence density, not repetition.
A diagnostic example
Suppose an employee asks, “Can I store customer exports in my project drive?” The generated answer is wrong.
A disciplined investigation asks:
- Did the connector ingest the current data-handling policy?
- Did parsing preserve the exception table?
- Did the chunk include the definition of “project drive”?
- Did hybrid search retrieve both the rule and its exception?
- Did reranking prefer the current policy over an old FAQ?
- Did security trimming remove evidence the employee was allowed to see?
- Did context assembly label version and effective date?
- Did the model stay within the evidence?
- Did the citation point to the exact supporting section?
Only the eighth question is primarily about generation. Most RAG failures happen earlier.
Treat abstention as a successful outcome
A system that says “I cannot find an approved source for that exception” may be performing better than one that invents a plausible policy. Define evidence thresholds, conflict rules and questions that require escalation to a human owner.
Measure unsupported-answer rate and useful-abstention rate separately. If the system abstains too often, improve coverage and retrieval. If it answers too freely, tighten the evidence contract. Do not optimise only for the number of questions receiving an answer.
RAG earns trust when absence remains visible.