In this module9 lessons
A/B tests a PM must be able to read
Read an experiment result correctly, covering p-values, intervals, power, peeking, novelty, guardrails, interference and Simpson's paradox.
7 min readFull kit
Why PMs get asked about experiments
Many product teams decide launches with A/B tests. Interviewers want to know whether you would make a sound call from a result or a confident wrong one. You won't be asked to derive formulas. You will be asked to read a result and say what you would do.
Checking your access…
Practice this lesson
- An A/B test shows a 3% lift in clicks but a 1% drop in purchases. Do you ship?
- Your A/B test is significant on day two. Should you stop it and ship?
- How would you A/B test a new pricing page?
- Explain what a p-value means to a stakeholder who has never taken statistics.
- What is a guardrail metric, and which ones would you set for a test that changes news feed ranking?