← All questions
Metricswarm-up
Explain what a p-value means to a stakeholder who has never taken statistics.
How to approach it
Use a plain analogy: how surprised should we be by this result if the change had no effect at all?
How a strong answer opens
“A p-value answers one question: if this change had no effect at all, how surprising would results like ours be? A small p-value means very surprising.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Start with the question the p-value answers
- Use a simple analogy, such as a coin flip
- Explain the threshold and what it means
- Say what a p-value does not tell you (size of effect, importance)
- Connect it to the decision at hand
Where this question usually goes wrong
Saying the p-value is the chance the change works, or that a small p-value means the effect is large or important.
Learn the method
A/B tests a PM must be able to read in Module 2, Metrics and execution.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More metrics questions
- An A/B test shows a 3% lift in clicks but a 1% drop in purchases. Do you ship?
- Your A/B test is significant on day two. Should you stop it and ship?
- How would you A/B test a new pricing page?
- What is a guardrail metric, and which ones would you set for a test that changes news feed ranking?
- Weekly active users are up, but revenue is down. How do you explain this to leadership?