← All questions
Metricsstandard
An A/B test shows a 3% lift in clicks but a 1% drop in purchases. Do you ship?
How to approach it
Go back to the goal: clicks are a means, and purchases are closer to the outcome. Then check whether the purchase drop is statistically real.
How a strong answer opens
“Clicks are a means and purchases are closer to the goal, so the purchase drop worries me more. First I'd check whether both changes are statistically real.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Restate the test's goal
- Check significance and sample size for both metrics
- Consider which metric matters more to the business
- Explore why clicks rose but purchases fell (curiosity clicks, worse targeting)
- Look at segments where it worked or failed
- Recommend: if the purchase drop is real, do not ship as is; iterate on the variant
Where this question usually goes wrong
Shipping because the primary metric won, or killing the test without checking whether the purchase drop is significant.
Learn the method
A/B tests a PM must be able to read in Module 2, Metrics and execution.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More metrics questions
- Your A/B test is significant on day two. Should you stop it and ship?
- How would you A/B test a new pricing page?
- Explain what a p-value means to a stakeholder who has never taken statistics.
- What is a guardrail metric, and which ones would you set for a test that changes news feed ranking?
- You shipped a new onboarding flow for a photo-editing app. How do you know it is working?