← All questions
Metricsstandard
What is a guardrail metric, and which ones would you set for a test that changes news feed ranking?
How to approach it
Define a guardrail as a metric you do not want to harm while chasing the main goal. Then list ones a ranking change could hurt.
How a strong answer opens
“A guardrail is a metric you agree not to harm while chasing the main goal. For a feed ranking test, I'd name the primary metric first, then the things ranking could quietly damage.”
Outline of a strong answer
Try answering out loud for a few minutes before you open this.
- Define guardrail metrics in one sentence
- Name the primary metric for the ranking test
- Guardrails: content creator reach, report rate, hide rate
- Long-term: retention, time spent quality
- Business: ad revenue and advertiser results
- Explain how guardrails affect the ship decision
Where this question usually goes wrong
Listing guardrails without saying how they change the ship decision, or forgetting creators and advertisers who are affected by ranking.
Learn the method
A/B tests a PM must be able to read in Module 2, Metrics and execution.
The full kit has worked answers in full, start to finish, for every round type. $129 once.
More metrics questions
- An A/B test shows a 3% lift in clicks but a 1% drop in purchases. Do you ship?
- Your A/B test is significant on day two. Should you stop it and ship?
- How would you A/B test a new pricing page?
- Explain what a p-value means to a stakeholder who has never taken statistics.
- Checkout conversion on an e-commerce site dropped after a release. Walk me through your diagnosis.