← All questions
Metricsstandard
Your A/B test is significant on day two. Should you stop it and ship?
How to approach it
Explain the risk of peeking: checking early and often makes false positives more likely. Then mention novelty effects and weekly cycles.
How a strong answer opens
“I wouldn't stop yet. Checking early and stopping as soon as a result looks significant makes false positives much more likely, and two days misses weekly patterns.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Explain why stopping early inflates false positives
- Mention weekly cycles: weekday users differ from weekend users
- Mention novelty effects: users click new things just because they are new
- Check the planned sample size and duration
- Recommend running to plan unless there is harm
- Note when early stopping is fine (harm or a design built for it)
Where this question usually goes wrong
Saying yes because the result is significant, without knowing that peeking inflates false positives and that novelty effects fade.
Learn the method
A/B tests a PM must be able to read in Module 2, Metrics and execution.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More metrics questions
- An A/B test shows a 3% lift in clicks but a 1% drop in purchases. Do you ship?
- How would you A/B test a new pricing page?
- Explain what a p-value means to a stakeholder who has never taken statistics.
- What is a guardrail metric, and which ones would you set for a test that changes news feed ranking?
- How would you measure the success of Amazon's 'Buy it again' feature?