← All questions
Metricsstretch
How would you A/B test a change in a marketplace where buyers and sellers affect each other?
How to approach it
Name the problem: test and control groups share the same supply, so changes leak between them. Then suggest designs that limit the leak.
How a strong answer opens
“The tricky part is that test and control buyers draw on the same sellers, so a change in one group leaks into the other. I'd pick a design that limits that leak.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Explain interference between groups
- Give an example: a buyer discount drains supply from control buyers
- Options: split by region or city, by time window, or by cluster
- Discuss trade-offs: fewer units, less statistical power
- Pick an approach for this case
- Define metrics on both sides of the market
Where this question usually goes wrong
Running a normal user-level split and trusting the result, without noticing that treatment users can take supply away from control users.
Learn the method
Worked cases 6 to 10 in Module 2, Metrics and execution.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More metrics questions
- What metrics would you use to measure the success of LinkedIn's 'People you may know'?
- How would you measure whether Google Maps' traffic predictions are good?
- What metrics matter most for a B2B project management tool?
- You shipped a new onboarding flow for a photo-editing app. How do you know it is working?
- How would you measure the success of Amazon's 'Buy it again' feature?