What AI product rounds test

Not long ago, an AI question in a PM interview was a curiosity. Now it is common. Some companies run a dedicated AI product round. Many more slip AI into rounds you already know: a product sense prompt that says "use AI", a metrics question about a chatbot, a strategy question about whether to build on a third-party model.

The good news is that these questions reward the same habits as every other PM round: a clear user, a sharp goal, honest trade-offs. The difference is that AI products fail in ways ordinary software does not, and interviewers want to see that you know those ways.

This lesson covers where the questions show up, what interviewers are listening for, and the one method you will use for every answer in this module.

#Where AI questions show up

Expect four shapes. The wording varies, but the shape is usually easy to spot within the first sentence.

A fifth shape appears in technical rounds: "Explain how a large language model works" or "What is retrieval-augmented generation?" You do not need to be an engineer. You need to explain the ideas plainly and connect each one to a product decision. The next lesson, How large language models work, for PMs, gives you exactly that depth.

#What interviewers are listening for

You may not see a separate rubric for AI questions. Expect the usual criteria: structure, user focus, judgment, communication. But AI prompts expose five specific signals, and strong candidates hit all five without being asked.

1. Fit. You ask whether AI is the right tool before designing with it. A candidate who bolts a chatbot onto every prompt signals hype, not judgment.
2. Failure awareness. You know the model will sometimes be confidently wrong, and you design for that: review steps, fallbacks, undo, a clear "I'm not sure".
3. Evaluation literacy. You can say how you would test quality before launch and track it after. "We'll A/B test it" is not enough when you cannot yet say what "good" means.
4. Cost and speed realism. You know every response costs money and takes time, and that a bigger model is not free.
5. Risk and trust. You raise privacy, bias, abuse and prompt injection at the right moment, in proportion to the product, not as a checklist recited at the end.

Notice what is missing: model architecture, training details, benchmark scores. Interviewers almost never reward those in a product round. A candidate who spends three minutes on how attention layers work has used three minutes not designing a product.

Key idea In an AI product question, the most important design decision is what happens when the model is wrong. Candidates who design only the happy path sound like they have never shipped one.

#The method: the Five Gates

This module uses one method for every AI question. It is called the Five Gates because an idea has to pass through each one before you would ship it. Each gate is a question you answer out loud.

1. Job. Who is the user, what task are they trying to get done, and what does a good outcome look like for them?
2. Fit. Is AI the best tool for this job, and which kind of AI capability does it need: generating, summarizing, extracting, classifying, searching or taking actions?
3. Failure. When the model gets it wrong, what happens, and how does the design catch, show or recover from it?
4. Proof. How will you know it works: what you test before launch, and what you measure after?
5. Ship. What will it cost, how fast must it be, what could go badly wrong, and how will you roll it out?

The gates sit on top of the product sense method from The framework in one page, not in place of it. You still clarify, pick a user and prioritize. The gates add the questions that are specific to AI, in the order an experienced AI PM would ask them.

For execution questions, such as a metric drop, you walk the gates in reverse: ask what changed in what you shipped (gathering facts, not blaming yet), confirm the measurement before concluding anything, look for new failure patterns, check whether users or their inputs changed, and finally ask whether the job itself changed. The Worked answers lesson shows both directions in full.

#The gates in thirty seconds

Here is how the gates sound on a small prompt: "Add AI to a recipe app."

A weak opening goes straight to a feature: "I'd add a chatbot where you can ask it anything about cooking." It might be fine. But the interviewer has learned nothing about your judgment.

A strong opening sounds like this:

You Before I pick a feature, I'll run it through five gates: the job, whether AI fits it, what happens when it's wrong, how we'd prove it works, and what shipping it costs. On the job: I'll focus on a home cook on a weeknight who has some ingredients and thirty minutes. Their job is "get a decent dinner from what I have." On fit: matching loose ingredient lists to recipes, and adapting a recipe when something is missing, is a fuzzy language task where AI does well and rules do badly. Plain search already handles "show me lasagna recipes", so I wouldn't use AI there. On failure: the risky case is a bad substitution, or worse, one that ignores an allergy. So allergies become hard filters applied outside the model, and substitutions show a short reason the cook can judge. I'll come back to proof and shipping once we pick the feature.

That took under a minute. It named a user, chose where AI earns its place and where it does not, and caught the one failure that could hurt someone. That is the bar.

Try it Take the prompt "Add AI to a fitness app." Say the first three gates out loud in under ninety seconds. Did you name one place you would not use AI? If not, try again.

#How this module is organized

Each lesson goes deeper on one gate or one skill:

• How large language models work, for PMs: tokens, context windows, temperature, hallucination, retrieval, fine-tuning and agents, each tied to a product decision.
• Designing an AI feature: the Job, Fit and Failure gates, including when not to use AI and how to show uncertainty.
• Evals and quality: golden sets, rubrics, model-graded evals and their biases, and regression testing.
• Metrics for AI products: what to measure once real users arrive, and which metrics lie.
• Risk, safety and trust: privacy, bias, abuse and prompt injection, in proportion.
• Cost and latency trade-offs: model size, caching, batching and routing.
• Worked answers: four full answers in dialogue form.
• Common traps and how to recover: the mistakes that cost the most points.

If you have a week, read this module after Product sense and design and Metrics and execution. The AI gates assume you can already run a design answer and break down a metric.

#Recap

• AI questions come in four main shapes: design, execution, evaluation, and judgment. A fifth, explaining the technology, shows up in technical rounds.
• Interviewers listen for fit, failure awareness, evaluation literacy, cost realism and proportionate risk thinking.
• The Five Gates are Job, Fit, Failure, Proof and Ship. Walk them forward for design questions and in reverse for diagnosis.
• The single most telling move is designing for the wrong answer, not just the right one.