How PMs can turn an AI eval into a product requirement before they build
Turn an AI feature idea into a testable product requirement with one user job, a measurable outcome, boundary cases, and a review rule before building expands.
Niels Kaspers
TL;DR
An AI feature becomes a product requirement when a team can name the user job, observable outcome, unacceptable boundary case, and review rule before it starts optimising a prototype.
An AI product requirement is not “add a copilot” or “use an agent to automate this.” It is a statement that a specific person can complete a specific job to an observable standard, with known limits on when the system must stop, ask, or hand work back. If a team cannot write that down, it does not yet have a requirement. It has a promising demo.
That distinction matters because AI prototypes create conviction quickly. A team sees a model produce one impressive answer, then starts discussing surfaces, integrations, and rollout. The hard product questions arrive later: Which user decision got better? What evidence counts as success? What happens when the model is uncertain, missing context, or wrong in a high-consequence case?
I use a small requirement packet before expanding an AI workflow: one user job, one observable outcome, a boundary-case set, and a review rule. It is deliberately smaller than a traditional specification. Its job is to make the feature testable before the implementation becomes expensive to change. That is the same discipline behind defining an AI prototype’s one real user job: capability only becomes a product when it is anchored to an outcome a user can recognise.
Start with the decision, not the model capability
The requirement should name the decision or task that changes for a user. “Summarise customer feedback” describes model activity. “Help a PM decide whether a repeated support issue warrants a discovery call this week” describes a user job. The difference forces the team to identify the input, the person responsible for the call, and the consequence of a poor result.
Write the user job in one sentence: When [context], [user] needs to [make a decision or complete a task] so that [outcome]. Then remove anything the model can do that does not help that sentence. This is a useful constraint for small teams because AI makes scope expansion cheap enough to be dangerous.
The first-party lesson from building product and editorial workflows on NielsKaspers.com is simple: a system earns autonomy by returning a reviewable result for a bounded job. An automation can collect sources, draft a candidate, and surface links; it should not be judged by the fact that it generated text. It should be judged by whether the reviewed output helps the publisher make a better, faster decision.
Turn the job into an observable outcome
Next, describe the result in a way that someone other than the builder can inspect. “Useful” is not a metric. “The PM can identify the three repeated issues, open the underlying evidence, and assign a next research action in under ten minutes” is observable.
This does not mean every AI feature needs a perfect numeric target before launch. It means the team needs a shared test for whether the job happened. OpenAI’s agent-evaluation guidance recommends inspecting traces while debugging behaviour, then moving to repeatable datasets and eval runs once the team knows what good looks like. Product requirements need the same sequence: inspect real examples first, then formalise the recurring standard.
A practical outcome has three parts:
- Result: what the user can now decide, create, or complete.
- Evidence: what lets them verify the result rather than trust a fluent answer.
- Cost: the time, review effort, error risk, or workflow friction that must improve.
For example, a research assistant requirement might be: “For a supplied set of interviews and tickets, produce a ranked evidence table that a PM can trace to source excerpts and use to select one next validation step.” The acceptance test is not that the summary sounds thoughtful. It is that the evidence table supports the selection.
Define the boundary cases before the happy path wins
The most expensive AI surprises are usually not failures to produce an answer. They are cases where the answer appears usable but should not be acted on: conflicting evidence, missing account context, a policy-sensitive action, stale data, or an unsupported claim.
Put a small boundary-case set in the requirement before the first broad rollout. Three to five representative cases are usually enough to start: one ordinary case, one ambiguous case, one missing-context case, one high-consequence case, and one case where the correct behaviour is to decline or escalate.
Anthropic’s guidance on agent evals makes the relevant product point: people closest to the requirements and users are best placed to define success. Do not ask a model team to invent all of these cases in isolation. Bring in the PM, support lead, operations owner, or domain expert who knows what a deceptively plausible failure looks like.
This is also where deciding what should stay human in an AI workflow becomes a product requirement rather than an abstract principle. A boundary case should say who owns the final decision and what evidence they need when the system cannot earn that autonomy.
Interactive
AI product requirement check
Use this before expanding an AI prototype beyond a narrow test.
Completion
This is the gap between understanding the article and actually using it.
- Use this block as the practical summary, not just the article ending.
- If one item feels vague, the article probably needs sharper guidance.
- A short checklist beats a long recap when the reader needs to act.
Add a review rule, not a vague human-in-the-loop promise
“Human in the loop” is too imprecise to guide a product. A review rule says when a person intervenes, what they see, and what they can do next. For a low-risk drafting tool, that might mean the user approves every external send. For a research workflow, it might mean the PM reviews only recommendations with weak evidence, conflicting segments, or an unfamiliar pattern.
The goal is not to put a person behind every output forever. It is to make oversight proportional to consequence and uncertainty. OpenAI’s macro-eval example describes looking for recurring behaviour across traces, including which business function owns the pattern. That is the useful follow-on: a review rule should produce signals that reveal whether the feature needs a product change, a data fix, a workflow change, or a clearer ownership boundary.
A good rule is short enough to test: “Escalate when the evidence sources conflict, when the account is enterprise, or when the recommendation would trigger an irreversible change.” It creates a queue of decisions rather than a dashboard full of undifferentiated errors.
Treat the first eval set as a requirement artifact
Once the team has five to ten representative examples, store them with the requirement. They are not just engineering test data. They are the clearest expression of what the product promises. When the feature changes, rerun them. When a customer reports a consequential miss, decide whether it belongs in the set.
This prevents a familiar failure mode: a team ships an AI feature based on a few successful demos, then learns about edge cases only through production frustration. Turning AI failures into an eval backlog is useful after launch; writing requirement cases early moves some of that learning before the feature becomes a commitment.
The requirement can evolve. The point is not to freeze discovery. It is to give discovery a durable record of what was meant to work, for whom, under which conditions. That makes trade-offs visible when someone proposes a broader capability or a more autonomous workflow.
A compact template
Use this as a starting point:
- User job: When ___, ___ needs to ___ so that ___.
- Observable outcome: The user can ___, verified by ___.
- Success cost/risk: Improve ___ without increasing ___.
- Boundary cases: Ordinary ___; ambiguous ___; missing context ___; high consequence ___; escalate/decline ___.
- Review rule: A person reviews when ___. They see ___. They can approve, correct, retry, escalate, or stop.
- Initial eval set: The representative examples that must continue to meet the standard.
If the team cannot fill these fields without hand-waving, keep prototyping. But do not call the feature ready for a roadmap commitment yet. The missing detail is exactly where product risk is hiding.
FAQ
What is an AI product requirement?
It defines the user job, observable outcome, boundary cases, and review rule for an AI-assisted feature. It goes beyond describing a model capability by making the product promise testable.
How many eval cases should a product team start with?
Start with five to ten representative cases that include normal work, ambiguity, missing context, high consequence, and an escalation or refusal case. Add cases when real failures expose a meaningful gap.
Do AI product requirements need a quantitative metric?
Not always at the beginning. They do need an observable outcome and a way for a reviewer to tell whether the feature helped or harmed the user’s job. Quantitative targets can become stronger once the team has real examples.
Who should write the boundary cases?
The PM should coordinate them with people closest to the user and operational risk: support, operations, sales, design, engineering, or a domain expert. Their experience reveals failures that a happy-path demo will not show.