← All questions
AI productstandard
What makes a good eval set, and how would you build one for an AI travel-planning assistant?
How to approach it
A good eval set looks like real user requests, including the hard and risky ones, and comes with a clear definition of a good answer for each.
How a strong answer opens
“A good eval set looks like real user requests, including the hard and risky ones, with a clear definition of a good answer for each. For a travel planner, that means feasible plans with no invented places.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Gather real or realistic requests: weekend trips, family trips, tight budgets, accessibility needs
- Add edge cases: visa rules, closed attractions, impossible timings, vague requests
- Write a scoring guide: feasible plan, fits budget, no invented places or prices
- Score with trained reviewers plus automatic checks such as verifying that places exist
- Use it before every model or prompt change to catch regressions
- Refresh it from new user complaints and failure cases
Where this question usually goes wrong
Building an eval set of easy, happy-path requests the model already handles, so it never catches real failures.
Learn the method
Evals and quality in Module 7, AI product interviews.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More ai product questions
- How would you evaluate whether an AI summarization feature in an email app is good enough to launch?
- What metrics would you use for an AI writing assistant in a document editor?
- Your company wants an AI assistant that answers employee questions from all internal documents. What could go wrong, and how would you prevent it?
- How would you decide whether to use an off-the-shelf model API or train your own?
- Your AI feature costs about four cents per request and users love it, but it would lose money at scale. What are your options?