← All questions
Metricsstandard
How would you A/B test a new pricing page?
How to approach it
Clarify what changes on the page (layout or actual prices), because testing different prices raises fairness and trust issues.
How a strong answer opens
“I'd like to know what changes: the layout or the actual prices. Showing different prices to different people raises fairness and trust issues, so the test design depends on that answer.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Clarify the change and the hypothesis
- Pick the primary metric: paid conversion or revenue per visitor
- Add guardrails: refunds, support tickets, churn later
- Decide the unit (user, not session) and run time
- Discuss risks of showing different prices to different people
- Describe how you would read results and decide
Where this question usually goes wrong
Testing price changes like a button color, ignoring that customers compare prices and that refunds and churn show up weeks later.
Learn the method
A/B tests a PM must be able to read in Module 2, Metrics and execution.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More metrics questions
- An A/B test shows a 3% lift in clicks but a 1% drop in purchases. Do you ship?
- Your A/B test is significant on day two. Should you stop it and ship?
- Explain what a p-value means to a stakeholder who has never taken statistics.
- What is a guardrail metric, and which ones would you set for a test that changes news feed ranking?
- Define success metrics for a video calling app's new noise cancellation feature.