How would you reduce hallucinations in an AI feature that answers questions from a company's help docs?
How to approach it
Explain why the model makes things up (it predicts plausible text, not checked facts), then show how grounding it in the help docs and allowing 'I don't know' cuts that risk.
How a strong answer opens
“The model makes things up when it lacks the facts and fills the gap with plausible text. So I'd retrieve the right help articles for each question, make it answer only from them, and let it say it doesn't know.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Explain the cause: the model fills gaps with plausible text when it lacks the facts
- Retrieve the most relevant help articles for each question before answering
- Instruct the model to answer only from those articles and cite them
- Let it say it does not know and hand off to a person
- Keep the docs current, since stale articles produce confident wrong answers
- Test with tricky and out-of-scope questions; track unsupported claims and thumbs-down rate
Where this question usually goes wrong
Offering a bigger model or a stricter prompt as the whole fix, without retrieval, citations, an "I don't know" path or a test set.
Learn the method
How large language models work, for PMs in Module 7, AI product interviews.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More ai product questions
- Explain how a large language model works to a non-technical executive.
- Design an AI feature for a recipe app.
- Design an AI assistant that helps customer support agents answer tickets faster.
- How would you evaluate whether an AI summarization feature in an email app is good enough to launch?
- What metrics would you use for an AI writing assistant in a document editor?