Metrics Design
Metric selection determines what the organization optimizes. Selecting an inappropriate metric leads to optimizing the wrong outcome.
Metric Quality Criteria
A useful metric must satisfy five requirements:
| Criterion | Question | Failure Example |
|---|---|---|
| Measurable | Can it be computed from available data? | "User delight" without operationalization |
| Movable | Can product changes affect it? | "Total users ever" (only increases) |
| Unambiguous | Does everyone agree on the definition? | "Active users" without defined actions |
| Timely | Is it available quickly enough for decisions? | Customer lifetime value (requires years) |
| Robust | Is it resistant to gaming? | "Time on page" (can be inflated with loading delays) |
Metric Categories
North Star Metric
The single metric that captures whether the product delivers value to users.
| Company | North Star Metric | Rationale |
|---|---|---|
| Netflix | Hours watched | Indicates value delivered |
| Airbnb | Nights booked | Represents completed transactions |
| Slack | Messages sent | Measures active usage |
| Uber | Rides completed | Core value delivery |
| Spotify | Listening time | Engagement depth |
The north star metric provides organizational alignment. When priorities conflict, teams evaluate which approach has greater impact on the north star.
Primary Metrics
For specific experiments or features, one metric determines success.
| Context | Primary Metric |
|---|---|
| Checkout flow redesign | Conversion rate |
| Onboarding experiment | Time to first action |
| Pricing test | Revenue per user |
Multiple primary metrics create ambiguity when results conflict.
Guardrail Metrics
Metrics that should not degrade while optimizing the primary metric.
| Category | Example Guardrails |
|---|---|
| Performance | Load time, error rate |
| Revenue | Revenue (when optimizing engagement) |
| User satisfaction | Support tickets, complaint rate |
| Retention | Return rate, churn |
A 10% conversion lift that doubles page load time may represent a net negative.
Counter Metrics
Metrics that specifically prevent gaming of the primary metric.
| Primary Metric | Counter Metric | Rationale |
|---|---|---|
| Engagement (time spent) | Content quality, satisfaction | Prevents low-quality addictive content |
| Revenue | Churn rate, complaints | Prevents aggressive monetization |
| Conversion rate | Average order value, return rate | Prevents misleading conversions |
| Click-through rate | Actual conversions, bounce rate | Prevents clickbait |
HEART Framework
Google's framework for product metrics:
| Category | Measurement Focus | Examples |
|---|---|---|
| Happiness | User satisfaction | NPS, survey scores, app ratings |
| Engagement | Usage intensity | DAU/MAU ratio, session length, actions per session |
| Adoption | New user uptake | Sign-ups, feature activation, first-time use |
| Retention | User persistence | D7/D30 retention, churn rate, repeat usage |
| Task Success | Completion efficiency | Completion rate, time to complete, error rate |
Not all categories apply to every feature. Select relevant categories based on the feature's purpose.
Common Interview Topics
Feature Success Measurement
Framework:
- Identify the feature goal
- Select primary metric (directly measures goal)
- Define guardrail metrics (should not degrade)
- Specify data sources and calculation methods
New Product Metrics
Apply HEART framework or similar structured approach. Demonstrate consideration of multiple measurement dimensions, not just a single number.
Metric Gaming Prevention
Use counter metrics, diverse metric sets, and qualitative verification. Reference Goodhart's Law: "When a measure becomes a target, it ceases to be a good measure."
Leading vs Lagging Indicators
| Type | Characteristics | Example |
|---|---|---|
| Leading indicators | Predict future outcomes; enable early action | Engagement (predicts revenue) |
| Lagging indicators | Confirm outcomes after occurrence; higher accuracy | Revenue |
Both types serve different purposes: leading indicators provide rapid feedback; lagging indicators confirm actual business impact.
Metric Trade-offs
| Choice | Advantage | Disadvantage |
|---|---|---|
| Simple metric | Easy to understand and align on | May miss nuance |
| Composite metric | Captures multiple factors | Harder to interpret and debug |
| Short-term metric | Fast feedback cycle | May not reflect true impact |
| Long-term metric | Accurate outcome signal | Slow learning |
| Proxy metric | Available immediately | May not correlate with true goal |
| Direct metric | Measures actual outcome | May take too long to observe |
Each metric choice involves trade-offs. These trade-offs should be explicit and documented.