AI evals for product managers are repeatable tests that check whether an AI system does its job to a defined standard. The PM defines what "good" means for the task, builds a test set that mixes common cases, edge cases, and adversarial ones, chooses how each criterion is scored (code checks, human review, or a validated LLM judge), analyses the failures, and sets release thresholds by risk. The output of an eval is never just a score. It is a decision to ship, revise, limit scope, add human review, or stop.
Key takeaways
- Three polished examples are a demo, not an evaluation. Write the pass criteria before you run anything.
- Usage metrics and evals answer different questions. High adoption can hide confident wrong answers.
- Track severe failures separately. An invented refund promise must never disappear inside an average score.
- Use the simplest reliable evaluator: code for anything exact, humans for judgment, and LLM judges only after you validate them against human labels.
What are AI evals?
An eval is a repeatable test of an AI system against defined expectations. It can measure a classifier, a prediction model, a language model's output, a retrieval pipeline, an agent workflow, a guardrail, or a whole product experience. Evals help teams compare versions, expose failure modes, catch regressions, set launch thresholds, and monitor quality after release.
Evaluation is harder for AI than for normal software. A form either submits or it does not. An AI system can give a strong answer to one input and a weak answer to a nearly identical one. A prompt change can make answers more concise and quietly less complete. A new model can reason better and cost twice as much. A retrieval tweak can fix common questions and break a rare policy exception nobody tested.
That is why "this seems good" after looking at three examples is not evaluation. A disciplined team defines the task, decides what good means, builds representative test cases, picks scoring methods, runs the system consistently, studies failures, and ties the evidence to a product decision.
Why AI evals for product managers matter more than any other skill
Product analytics and evals answer different questions. Analytics tells you whether people use, return to, and pay for a feature. Evals tell you whether the AI performs its task well enough to deserve that use. The two are connected, but neither proves the other.
A support assistant is used by every agent every day. Usage cannot tell you whether its summaries are accurate or whether its replies promise refunds the policy does not allow. A knowledge assistant answers thousands of questions while citing documents that do not support its claims. A writing assistant generates hundreds of drafts that users rewrite from scratch. All three look healthy on a dashboard.
User feedback does not rescue you either. A confident wrong answer can earn a thumbs up from a user who cannot spot the error, while a correct answer can get a thumbs down because the interface was slow. You need behavioural data, output evaluation, expert review, and user research working together, and the PM is the person who designs that system. If the terms are new, the AI PM glossary defines them. It is why evals sit at the centre of the 22 AI PM skills and come up in almost every AI PM interview loop.
AI evals vs benchmarks vs A/B tests
| Method | What it answers | When to use it |
|---|---|---|
| Public benchmark | How a model performs on a generic, standardised task | Shortlisting models. It says little about your users or data. |
| Offline product eval | How your system performs on your task, with your data and your definition of good | Before every meaningful change to model, prompt, retrieval, or tools |
| Online eval | How the system behaves on real, messy production traffic | Continuously after launch, sampled by risk |
| A/B test | Whether a change causes a better user or business outcome | When you have enough traffic and quality already clears the bar |
A model that tops a leaderboard can still fail your refund policy questions. Benchmarks help you choose what to try. Product evals decide what you ship.
The AI eval loop in six steps
- Define success for one narrow task, including which errors are severe.
- Build the test set mixing common cases with edge cases and adversarial ones, plus the expected behaviour for each.
- Choose evaluators per criterion: code, human, or a validated model judge.
- Run and analyse the results by case type and segment, then group failures into categories.
- Decide against thresholds you set in advance: ship, revise, narrow, add review, or stop.
- Monitor production and feed confirmed failures back into the test set.
The rest of this guide on AI evals for product managers walks through each step with examples you can copy.
Step 1: Define "good" with a rubric
"Helpful" is not a criterion. Helpful for which user? Supported by which source? Complete according to which policy? Safe within which boundary? Answering those questions is the PM's job, and it happens before anyone scores an output.
Pick the dimensions that matter for this workflow
| Dimension | The question it asks |
|---|---|
| Faithfulness | Is every claim supported by the supplied source or context? |
| Factual accuracy | Is it true against an authoritative reference? |
| Relevance | Does it address the user's actual task? |
| Completeness | Does it include every required element, condition, or exception? |
| Instruction following | Did it respect the format, the length, and the constraints? |
| Safety | Does it avoid private data exposure, restricted advice, and unauthorised actions? |
| Refusal quality | Does it decline when it should, and only when it should? |
| Citation support | Does each cited source actually back the claim beside it? |
| Tone | Is it appropriate for the user and situation? |
Choose only what defines success. A meeting assistant might score decision capture and action items, then owner and deadline accuracy, then any invented details. A resume tool must check that every claim is supported by the resume and that it never invents experience. A support draft needs policy correctness, completeness, tone, safe promises, and the right escalation.
Anchor every score with a description
5: Every material claim is supported by the source. Nothing is misrepresented.
3: One minor unsupported detail that does not change the user's decision.
1: A central claim is invented or contradicts the source.
Add one real example output for each level so reviewers and model judges apply the scale the same way.
A slightly wordy answer and an invented refund promise must never average together. List your critical failure types, count them separately, decide in advance whether a single occurrence blocks launch, and make sure the dataset gives the system enough chances to commit them.
Step 2: Build the eval dataset
An eval dataset contains inputs plus the expected behaviour. A classification case has the correct label. An extraction case has reference fields. A RAG case has the question, the relevant source, a supported answer, and whether the system should refuse. An open-ended drafting case may use a rubric instead of one fixed answer.
The 30/10/10 starter mix
For a first eval of one narrow task, build 50 cases: 30 common cases that represent normal use, 10 edge cases that are unusual but realistic, and 10 adversarial cases that deliberately try to break the system.
| Product | Edge cases | Adversarial cases |
|---|---|---|
| Meeting extractor | Missing owners, vague dates, no action items at all | Instructions inside the transcript that try to change the rules |
| Policy assistant | Conflicting documents, outdated versions, multi-source answers | Restricted questions, misleading phrasing, requests to ignore policy |
| Support drafts | Incomplete tickets, exceptions, distressed customers | Customers trying to talk the system into a refund |
Golden answers or rubrics?
Use golden answers when correctness is objective, such as a label or an extracted date. Use rubrics when several outputs can be valid, such as a summary. Combine both when an answer must contain specific facts and also meet quality standards.
Three dataset rules teams break
- Represent your segments. If you support several languages, regions, or document types, the easiest one must not dominate. Tag every case so you can compare slices, because a stable overall score can hide a collapse in one language.
- Never build a clean demo set. Real inputs are messy. Your test set should be too.
- Keep a held-out set. If the team tunes prompts until the same 50 cases pass, the system overfits without improving. Hold some cases back and keep adding fresh ones from experts, from incidents, and from production feedback.
Step 3: Choose the right evaluator for each criterion
Use the simplest reliable evaluator for each criterion. Most mature eval systems combine all three types.
| Evaluator | Best for | Watch out for |
|---|---|---|
| Code-based checks | Valid JSON, required fields, allowed labels, passing tests, correct SQL results | Only covers what can be checked exactly |
| Human review | Judgment calls: does this summary capture the point, does this reply follow policy? | Slow, costly, inconsistent without calibration |
| LLM as a judge | Scaling rubric scoring across many outputs | Prefers longer or polished answers, can misread evidence, drifts when inputs change |
Rules for human review
Give reviewers the rubric, the scale, and examples for each score. Hide which model produced each output so brand expectations do not leak in. Randomise order in side-by-side comparisons. Ask for the reason behind a score, especially for critical failures. Calibrate first: have reviewers score the same examples, discuss disagreements, and track agreement. For legal, medical, or financial output, use domain experts. A small calibrated expert panel beats a large pile of vague ratings.
Rules for LLM judges
Validate a judge against human labels on a representative set before trusting it, paying most attention to severe failures. Pairwise comparison or scoring against explicit criteria is usually more reliable than an open-ended critique. Give the judge the evidence it needs: a judge cannot check groundedness without the source. Never let the system being tested grade itself. Revalidate whenever the task, rubric, judge model, or input mix changes. The research paper Judging LLM-as-a-Judge documents the position and verbosity biases you are guarding against.
LLM evaluation metrics by product type
There is no universal score for AI products. Pick metrics through product risk, not habit.
Classifiers: precision, recall, and the accuracy trap
Take 1,000 support tickets. Twenty are genuine security incidents. A model that labels every ticket "ordinary" is right 980 times: 98% accuracy. It also misses all twenty incidents, which is the only job that mattered. A confusion matrix exposes the twenty false negatives instantly.
Precision asks: when the system flags something, how often is it right? High precision means fewer false alarms, which matters when blocking a legitimate payment damages trust. Recall asks: of all the real cases, how many did it catch? High recall means fewer misses, which matters when an undetected fraud or safety issue is costly. Raising one usually lowers the other as you move the decision threshold, so the product decides which mistake hurts more. Google's Machine Learning Crash Course has a clear interactive explanation.
Thresholds are product design. You might approve confident, low-risk cases automatically, send the uncertain middle band to human review, and block only the risky cases the model is sure about. Then measure the reviewer workload too, because a threshold that looks great in a notebook can bury a five-person team.
Regression models: MAE and RMSE
For predicted numbers such as delivery time or demand, MAE is the average size of the miss in real units ("five minutes off on average"). RMSE punishes big misses more heavily. Always look at the distribution and slice by region or segment. A delivery estimate with a fine average that is occasionally hours wrong is a trust problem, and being late usually costs more than being early.
The rest of the map
| Product type | What to measure |
|---|---|
| Summariser | Faithfulness, coverage, omissions, readability |
| Drafting assistant | Instruction following, factuality, tone, edit rate, acceptance |
| Coding assistant | Compiles, passes tests, security checks, follows conventions, developer acceptance |
| RAG assistant | Retrieval precision and recall at K, groundedness, citation support, refusal, usefulness |
| Recommendation system | Relevance, diversity, conversion, retention, fairness, long-term satisfaction |
| Agent | Task completion, tool-call correctness, step quality, safety, human interventions, recoverability, cost per successful task |
RAG needs retrieval and generation scored separately, explained fully in RAG for Product Managers. Agents need the path scored as well as the result, because an agent can write a perfect final message after updating the wrong account. That is covered in AI agents for Product Managers.
Step 4: Error analysis turns scores into fixes
A score is where analysis starts. If the system passes 78% of cases, the interesting part is the other 22%. Group every failure into a taxonomy with an owner and a likely fix.
| Failure category | Usually points to |
|---|---|
| Retrieval miss or wrong source | Search, metadata, or the data pipeline |
| Unsupported claim | Grounding instructions, evidence display, refusal rules |
| Missing information guessed | An output contract that allows "unknown" |
| Wrong classification | Labels, examples, or threshold |
| Unsafe or unauthorised response | Guardrails, permissions, approval steps |
| Format or instruction failure | Prompt, schema validation |
| Wrong tool or bad arguments | Tool definitions, argument validation, recovery |
| Correct but unusable | UX: clearer input, editable output |
A support system that invents refund promises may need better policy context and a hard approval boundary, not a longer prompt. A meeting tool that guesses owners needs permission to write "owner unknown." Error analysis is what stops a team from answering every problem with "try a bigger model." Hamel Husain's essay Your AI Product Needs Evals makes the same case with engineering examples.
Step 5: Set release thresholds by risk
Set thresholds before you see the results, or the thresholds will quietly move to match them. Define the minimum for internal testing, for a beta, and for general release, and list the failure types that block release regardless of the average.
| Risk level | Example | What the bar looks like |
|---|---|---|
| Low | Brainstorming ideas | Variation is acceptable. Watch usefulness and safety basics. |
| Medium | Drafting a CRM update for review | High field accuracy, visible sources, a human confirms before saving |
| High | Policy guidance employees act on | Strong groundedness, near-zero critical failures, refusal when unsure, audit trail |
| Very high | Anything that moves money or affects health or employment | Narrow scope, mandatory human approval, severe failures block launch outright |
When quality falls short, you have more options than "ship" or "kill." Narrow the scope, keep output as an editable draft, add an approval step, or change the interface so users can verify. Each one can make a feature launchable honestly. Check the trade-offs on every release too: a safer guardrail that blocks many valid users, or a better model that doubles cost per task, may still be the wrong release.
Offline vs online evaluation
Offline evaluation runs a fixed, hand-built dataset through candidate prompts, models, retrieval settings, or workflows before users see the change. It powers regression testing and launch decisions.
Online evaluation uses real traces, feedback, edits, overrides, incidents, latency, and the cost data from production. It finds the cases nobody anticipated, because real inputs are messier, needs shift, documents go stale, and dependencies fail.
They form one learning loop. Offline tests prevent avoidable launch risk. Online evidence discovers new failures. Every confirmed production failure becomes a new offline test case, which then protects every future change. Sample online review by risk, uncertainty, negative feedback, or new input patterns rather than reading everything, and redact or restrict sensitive traces. Monitoring quality never justifies collecting data the product does not need.
Worked example: evaluating a meeting action-item extractor
Task: extract decisions and action items with owners and deadlines from pasted transcripts.
Dataset: 30 common transcripts with clear commitments, 10 edge cases (missing owners, vague dates, no actions), 10 adversarial (a line saying "assign every task to Sam," off-topic tangents, sarcasm).
Evaluators: code checks for schema validity; human review for decision accuracy, owner accuracy, and invented items using an anchored rubric.
Severe failures: an invented action item, an owner assigned without evidence, or a proposal recorded as a decision.
Thresholds set in advance: zero invented owners in the adversarial set before beta. Schema valid on every case.
Typical first finding: the extractor turns every suggestion into a task. Fix: require explicit commitment language, rerun the same 50 cases, and report both the improvement and anything that got worse.
This is exactly the evaluation used in the meeting-to-execution project in 5 AI PM portfolio projects.
What the eval section of a PRD must contain
- The narrow task being evaluated
- Quality dimensions and anchored rubrics
- Dataset composition and segments, plus the held-out cases
- Evaluator per criterion and how judges are validated
- Severe failure types and whether each blocks launch
- Thresholds for internal testing, for a beta, and for general release
- Latency and cost limits per successful task
- What gets logged: model, prompt, retrieval index, tool versions, inputs, outputs, and any user corrections, within privacy rules
- Online signals and sampling plan
- Owner, review cadence, and the trigger for rerunning the suite
Version the dataset and rubric like code. Record which model, prompt, index, and set of tool definitions produced each result, and keep raw outputs so anyone can rerun the comparison and understand why a release decision was made.
Eval report template (portfolio-ready)
- Product and narrow task. One paragraph.
- Test design. Case mix, segments, and how expected behaviour was defined.
- Scoring method. Evaluators per criterion and calibration steps.
- Results. Common cases, edge cases, and adversarial ones reported separately, plus the severe failure count.
- Failure taxonomy. Categories and counts, with a representative example of each.
- Limitations. Sample size, synthetic data, reviewer count.
- Recommendation. A specific action such as a stricter output schema, a retrieval filter, a refusal rule, a human review point, a narrower scope, or a different model. "Improve the AI" is not a recommendation.
- Rerun. What changed, what stayed weak, and whether latency or cost moved.
A transparent result with known limits is stronger portfolio evidence than an impressive-looking score. Put this report in your case study, as described in how to build an AI PM portfolio.
Common AI eval mistakes
- Judging quality from a handful of impressive examples.
- Testing only the happy path.
- Reporting one aggregate score that hides severe failures and weak segments.
- Tuning endlessly on the same cases until the set is memorised.
- Treating evaluation as a one-time launch gate.
- Leaving latency and cost out of the comparison.
- Letting non-experts judge specialised output.
- Trusting an LLM judge that was never checked against human labels.
- Buying an expensive platform before building a first eval in a spreadsheet.
On tools: a spreadsheet with well-designed cases, recorded outputs, scores, and a failure category per row is a perfectly good first eval. Platforms such as the OpenAI Evals API, LangSmith, and Braintrust become valuable when you need repeatable experiments, trace analysis, and production monitoring across a team. Learn the workflow first. Vendors change.
AI evals for product managers in interviews
Two questions come up constantly. Here is how strong answers are structured.
"How would you evaluate a support assistant?"
Split it into layers. Product outcomes: agent adoption, handle time, resolution time, customer satisfaction. Output quality: summary accuracy, policy correctness, hallucinations, edit rate, escalation accuracy. Safety: unsafe responses and unauthorised promises, tracked as severe failures. Then describe the dataset mix, the evaluators, and the threshold that would block launch.
"Quality dropped after launch. What do you do?"
Check what changed first: model, prompt, retrieval index, documents, tools, guardrails, traffic mix, or deployment. Segment the decline by user type, by language, and by question category. Rerun the regression suite against the last good version. Read the failures and categorise them. That turns an alarming chart into a diagnosis, and it is exactly what interviewers want to hear. More practice questions are in AI Product Manager interview questions.
Questions & answers
8 questions readers ask most, answered straight.
What are AI evals in simple terms?
AI evals are repeatable tests that check whether an AI system does its job well enough. You define what good looks like for a specific task, build a set of test cases with expected behaviour, score the system's outputs, study the failures, and use the results to decide whether to ship, change, or limit the feature.
Why do product managers need to learn AI evals?
Because AI output varies and can be confidently wrong while usage metrics look healthy. The PM decides what quality means, which errors are unacceptable, and what bar a feature must clear before launch. Those are product decisions, so evaluation is a core product skill rather than a purely technical one.
How many test cases do I need for an AI eval?
Start with about 50 cases for one narrow task: 30 common cases, 10 edge cases, and 10 adversarial cases. Grow the set over time with production failures and expert examples, and keep some cases held out so the team does not tune the system to the test.
What is LLM as a judge?
LLM as a judge means using a language model to score another model's outputs against a rubric, a reference answer, or a side-by-side comparison. It scales review cheaply, but judges can favour longer or more polished answers, so validate them against human labels before trusting the scores.
What is the difference between precision and recall?
Precision measures how often the system is right when it flags something, so high precision means fewer false alarms. Recall measures how many of the real cases the system catches, so high recall means fewer misses. The product decides which matters more based on the cost of each mistake.
What is the difference between offline and online evals?
Offline evals run a fixed test set before a change reaches users, which supports regression testing and launch decisions. Online evals use real production traces, user feedback, and the incidents you log to find failures nobody anticipated. Confirmed online failures should be added to the offline set.
Can I do AI evals without coding?
Yes. A first eval can live in a spreadsheet with columns for input, expected behaviour, actual output, score, failure category, and notes. Code helps you automate and scale later, but the design of the cases and the rubric is where the real value sits.
What tools are used for AI evals?
Teams use spreadsheets for early evals, and platforms such as the OpenAI Evals API, LangSmith, and Braintrust for datasets, experiments, trace analysis, and monitoring. The workflow is the same across tools, so learn the method before choosing a vendor.
Where this comes from
This guide is condensed from chapters 37 to 40 of The AI Product Manager Blueprint by Abhishek Ashtekar (first edition, 2026). The book goes several levels deeper, with the full walkthroughs, templates, and examples.
External sources cited
- Your AI Product Needs Evals, Hamel HusainWhy teams need evidence stronger than demos, with error analysis examples.
- Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena, Zheng et al. (arXiv)Research on the strengths and biases of model judges.
- Evals guide, OpenAIStructured datasets, graders, and continuous evaluation.
- OpenAI Evals repository (GitHub)Open-source background on how an evaluation framework is organised.
- Accuracy, precision, and recall, Machine Learning Crash CourseInteractive explanation of classification metrics.
Last reviewed September 16, 2026. Tools, platforms, and salary data change; the book’s free resources page is updated as they move.
Browse all 88 chapters
This guide is the trailer.
The book is the whole system.
Everything this guide compresses, in full. The chapters, the skills, the portfolio projects, and the week by week roadmap that takes you from zero to hired.
- 88Chapters
- 22Skills
- 5Projects
- 1Roadmap
