RAG for Product Managers: retrieval-augmented generation explained, with failure modes and evals

What is RAG? Retrieval-augmented generation explained simply

Retrieval-augmented generation (RAG) is a pattern that gives a language model relevant external information right before it answers. Instead of relying only on what it learned in training, the system searches a set of approved sources, puts the useful passages into the prompt, and generates a response grounded in that evidence.

The simplest analogy is an open-book exam. A closed-book student answers from memory and sometimes confidently gets it wrong. An open-book student looks up the right page first, then writes the answer and can point to where it came from. RAG turns a model into the open-book student. The quality of the answer still depends on whether the student found the right page, read it correctly, and admitted when the book did not cover the question.

If terms like embeddings or tokens are new, keep the AI PM glossary open while you read. The idea was formalised in the 2020 paper Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks by Lewis and colleagues. Today it sits behind most company knowledge assistants, support copilots, and "chat with your documents" products.

Why RAG for product managers is a core skill

A model cannot reliably know private, current, or organisation-specific information unless the product supplies it. A support assistant needs this quarter's refund policy. An HR assistant needs the handbook for the right country. A sales assistant needs permitted account history. None of that is in any model's training data.

RAG earns its place because knowledge changes much more often than models are retrained. When the refund policy changes, the team updates the source and its index. Nobody retrains anything. Retrieval also enables citations, which make claims verifiable, and it connects a general model to a specific product domain.

When RAG is the right tool, and when it is not

That last row matters more than people think. Sometimes the right product decision is better search, not generation. AI PMs who say so in interviews stand out. If you are still building your foundations, the 22 AI PM skills guide shows where RAG sits among the other skills, and the zero-to-hired roadmap shows when to learn it.

How does retrieval-augmented generation work? Step by step

A RAG system has two paths. The indexing path runs before anyone asks a question. The query path runs every time someone does.

The indexing path

1. Collect approved sources. Policies, help articles, manuals, tickets, or transcripts. Remove drafts and duplicates.
2. Extract and clean the text. PDFs, tables, and any scanned pages often parse badly. Check samples by eye.
3. Chunk each document into retrievable passages.
4. Attach metadata to each chunk: title, owner, date, version, region, product area, and access level.
5. Create embeddings, numerical representations of meaning, and store them in a vector index alongside a keyword index.

The query path

1. Interpret the question. Clarify it, rewrite it into search terms, or split a multi-part question.
2. Apply permission and metadata filters so only documents this user may see, for the right region and version, are eligible.
3. Retrieve candidates with semantic search, keyword search, or both.
4. Rerank the candidates and keep the most relevant few.
5. Generate with instructions to use only the supplied evidence, cite claims, and say when evidence is missing.
6. Show the answer with sources and collect feedback.
7. Log a trace of sources, versions, prompt, model, latency, and the cost of each call so failures can be investigated.

RAG chunking explained for product managers

Chunking splits documents into pieces that can be found and placed in context. It sounds like an engineering detail. It often decides whether the product works.

A chunk that is too small might contain "this exception applies to annual plans" without the rule it modifies. A chunk that is too large drags in irrelevant text, raises token cost, slows responses, and distracts the model. Overlap between neighbouring chunks preserves context across boundaries but duplicates text. The right boundaries depend on the document type.

You do not choose chunk sizes. You ask whether the strategy matches the documents and the real questions. If an answer needs two clauses, can both come back? If a table carries the conditions, does it survive parsing? If the heading sets the scope ("EU customers only"), does the chunk keep it? Then insist the team compares chunking options on the same test questions instead of accepting a default. Anthropic's write-up on contextual retrieval is a good example of how much chunk context can change retrieval quality.

Embeddings, hybrid search, and reranking

Embeddings let search work by meaning. A user asks "how do I get my money back?" and the policy says "refund eligibility and reimbursement." Good embeddings place those close together, which is a huge advantage when users do not know official terminology. The OpenAI embeddings guide explains the idea well.

But similarity is not correctness. A passage can sound related without answering the question, and a semantically similar document can be the obsolete version. Exact error codes, product numbers, dates, and any legal clause references often need literal matching.

Hybrid search runs both. Keyword search catches "error E-442." Semantic search catches "my payment keeps failing." Most serious products benefit from having both paths.

Reranking is a second pass. The first search returns a wide set of candidates, then a reranker scores each against the question and reorders them, dropping plausible but unhelpful passages. It improves quality on large, messy collections and adds cost and latency. A 30-article FAQ may not need it. A legal policy workflow probably does. Cohere's rerank documentation shows how the second stage works.

Metadata, permissions, and the freshness of sources: the parts demos skip

Metadata turns similarity into product-aware retrieval

An employee asking about parental leave should get the policy for their country, employment type, and effective date. A customer should get answers for their product and plan. Region, version, status, and the access level often matter as much as meaning.

Permissions must be enforced before content reaches the model

An employee who cannot open an HR investigation file must never receive its contents through the assistant. Filter by access rights before retrieval, with the same rules as the rest of your product. When a request touches restricted material, refuse without revealing that the confidential document exists. Write this into the PRD: which sources each user type may access, where authorisation is checked, and how access is audited.

Freshness is a product requirement

Every important source needs an owner, a version, a review date, and an update interval. Daily indexing suits reference material. Prices, stock levels, and account state need a live tool call instead. A good support assistant retrieves the refund policy from documents and pulls the purchase date from the account system, and the PRD says which source wins when they disagree.

Citations and the "not found" state

Citations are RAG's strongest trust feature, but only when the cited passage actually supports the sentence beside it. If an answer says refunds are available within 30 days, the linked passage must say that, with its conditions. A source that merely discusses refunds creates false confidence, which is worse than no citation.

Match verification to risk. An internal knowledge assistant might show source titles and exact passages. A legal product may need clause-level attribution. A support tool might show evidence to the agent while the customer reply contains no internal citation at all.

Then design the honest states:

• No supporting evidence: "I could not find this in the approved documents," plus the next best step.
• Ambiguous question: ask one clarifying question instead of guessing.
• Outdated source: answer with a visible warning and the document date.
• Restricted content: decline without revealing what exists.
• Conflicting sources: show both and flag the conflict for the document owner.

A system that always produces a polished answer is less trustworthy than one that knows its limits. This is the same principle behind every strong eval plan, covered in AI evals for Product Managers.

RAG vs fine-tuning vs long context vs MCP

These four get confused constantly, including in interviews. They solve different problems and often work together.

"Train the model on our documents" is usually a retrieval need described with the wrong words. Fine-tuning helps once you have many representative examples of a behaviour you want. The Model Context Protocol can be the way a RAG system or agent reaches a data source, so "RAG vs MCP" is less of a choice than it sounds. A practical system might retrieve a relevant set, reason over it with a long-context model, and call a live tool for current account data, all in one answer.

Is RAG dead now that context windows are huge?

No. Long context works well for a user's own document. Sending an entire knowledge base on every question is expensive and slow, adds noise, and creates risk, because it can expose content the user should not see. Retrieval is selective by design, and selection is exactly what permissions and cost control need.

The 8 RAG failure modes and how to fix each one

Name the failure before you try to fix it. Changing the prompt cannot fix a document that was never retrieved.

Monitoring often shows that the real problem is the knowledge itself. If agents keep asking about an undocumented exception, the roadmap item might be a clearer policy, not a better model. RAG exposes weak knowledge management, and fixing the source often creates more value than any model change.

The four RAG trade-offs every AI PM must make

1. Retrieval quality versus cost and complexity. Better embeddings, richer metadata, hybrid search, reranking, and any answer validation each add infrastructure. A brainstorming tool and an employee policy assistant should not share a quality bar.
2. Latency versus accuracy. Live customer support needs answers in seconds, so fewer chunks, cached answers, and faster models make sense. Contract review can wait longer for stronger checks.
3. Indexed knowledge versus live data. Indexes are predictable and searchable by meaning. Live systems give exact current state but need permissions and error handling. Most good products use both.
4. Smooth answers versus transparent attribution. A fluent synthesis across five sources reads well and hides which source supports which claim. High-risk factual work should favour verification, revealed progressively so it does not overwhelm.

Cost and latency decisions like these are a skill of their own. The AI agents guide covers the same trade-offs once a system starts taking actions.

How to evaluate a RAG system

Evaluate at three connected levels. Skipping one hides where the problem lives.

Level 1: Retrieval quality

Did the right evidence appear, and near the top? Engineering teams use ranking metrics, and you should know what each one rewards:

• Precision at K: of the top K passages, how many were relevant. Rewards less noise.
• Recall at K: of all the passages needed, how many appeared in the top K. Rewards finding everything required.
• Mean Reciprocal Rank (MRR): how high the first correct passage ranks, on average.
• nDCG: rewards putting the most relevant passages first, with graded relevance.

Level 2: Answer quality

Was the answer grounded, correct, complete, clear, appropriately cautious, and correctly cited? Reviewers must check the source, not whether the prose sounds right. Track unsupported claims as hallucinations, and check refusal behaviour on questions with no answer.

Level 3: Product outcome

Did users finish the task faster or better without unacceptable risk? Track task completion, answers accepted or edited, source opens, error reports, escalations, latency, cost, and how often people come back. Interpret carefully: a source click can mean healthy verification or confusion, and a high acceptance rate can hide automation bias.

Build the test set before launch

Record the expected source and acceptable outcome for each case. Set release thresholds and severe-failure limits before you look at results. One permission leak can outweigh a hundred minor wording issues.

Worked example: a refund assistant for support agents

What goes in a RAG PRD: RAG for product managers in practice

A strong RAG PRD describes the knowledge system behind the interface. Use this as your checklist.

• Target user and workflow, plus a narrow launch scope
• Approved sources, their owners, and the update cadence
• Metadata fields and permission rules by user type
• Supported and explicitly unsupported questions
• Retrieval strategy: semantic, keyword, hybrid, reranking
• Citation behaviour and verification level
• Responses for missing answers, conflicting sources, and restricted content
• Latency and cost targets
• Evaluation set, release thresholds, severe-failure limits
• Analytics events, trace logging, and privacy boundaries
• Human handoff and escalation path

Start narrow. "Ask anything" makes the knowledge base, the risk, and the evaluation surface too broad. "Answer onboarding questions from the current employee handbook" is a release you can actually test. Expand only after retrieval, citations, permissions, and the user outcomes work in that first scope.

How to explain RAG in an AI PM interview

Interviewers ask "what is RAG?" to see whether you understand a product system or repeat a buzzword. Here is an answer structure that works.

RAG retrieves relevant information from approved sources, puts that evidence into the model's context, and generates a grounded answer with citations. Documents are chunked and embedded, then indexed with metadata. At query time we filter by permissions, retrieve with semantic and keyword search, rerank the results, and then generate. It fails in specific ways: missed or wrong evidence, stale sources, permission leaks, unsupported citations, over-answering, and prompt injection. So I would evaluate retrieval and answers separately on a realistic test set, set severity-based release thresholds, and monitor production traces.

Then be ready for follow-ups on chunking trade-offs, when you would not use RAG, and how you would handle two conflicting policies. More practice questions are in AI Product Manager interview questions.

Build a small RAG prototype this week

You do not need to be a retrieval engineer to learn RAG for product managers properly. Pick a document-question tool or low-code workflow that lets you upload sources, inspect citations, and replace documents.

1. Choose ten public help articles or a short product manual. No confidential data.
2. Write 20 test questions: direct, paraphrased, exact identifiers, multi-source, ambiguous, unanswerable, conflicting, and one that tells the assistant to ignore its instructions.
3. For each, record the expected source, retrieved source, answer, citation support, refusal behaviour, and latency.
4. Group the failures using the eight failure modes above.
5. Write a one-page evaluation report with the product changes you would make.

That report is portfolio-ready, and it grows into the full RAG knowledge-base project in 5 AI PM portfolio projects. The book's Chapter 34 goes deeper into every component, and Chapter 51 turns it into a complete case study.