8. Metrics inform, never set

§The original guide put it in one line and this chapter is built around it: metrics should inform strategy, not drive it. At a large company that is a caution. At a startup it is a survival rule, because the numbers are small enough to say anything.

§ 8.1Instruments, not steering#

§A metric is a question the team has already decided to ask, expressed as a number. Daily active users asks "how many people came back today?" Conversion asks "what fraction did the thing?" The number is only as good as the question, and the question is a product decision made before any data exists.

§This is why metrics cannot set strategy. Strategy chooses which questions matter. A team that lets the dashboard choose is letting last month's questions run next month's product, and at a startup last month's questions were guesses.

§Eric Ries's term for the decoration is the vanity metric: a number that goes up and feels like progress without telling you anything you could act on. Total signups. Page views. Cumulative anything. The test is simple. If the number went down tomorrow, what would you do differently? If the answer is nothing, it is vanity.

§ 8.2HEART, read at startup scale#

§Google's HEART framework was written for products with millions of users, and it survives translation to a product with forty because its categories are the right questions, not the right sample sizes.

§Happiness: do people like it? Measured by asking, not by inferring. Engagement: how much do they use it? Frequency and depth, per user, not in aggregate. Adoption: are new people starting? Retention: do they come back? Task success: can they finish the job? The framework's real gift is the discipline underneath it: for each category, name the goal, the signal that would show progress toward it, and only then the metric. Goal, signal, metric. Most startup dashboards skip the first two and wonder why the third means nothing.

§At five people, one metric per category is plenty and one metric overall is better. Rahul Vohra's account of building Superhuman describes a single question, how disappointed users would be if the product disappeared, used as the one instrument the whole team steered by for a year. The number was small, the sample was small, and it worked because the question was exactly the one the strategy needed answered.

§ 8.3Cohorts are the only honest view#

§Aggregate numbers lie at small scale. A product with two hundred users whose total usage rose twenty percent this month may have gained sixty new users and lost forty old ones, and those are opposite stories. The only way to see which is to follow groups of people who started at the same time and watch what they do week by week.

§That is a cohort, and Lean Analytics and The Lean Startup both treat it as the basic unit of product truth for exactly this reason. A cohort table answers the question aggregates hide: of the people who arrived in week one, how many are still here in week four? If the answer is roughly the same for every cohort, the product has not changed. If it is rising, something the team did is working. If it is falling, the growth number on the dashboard is a treadmill.

§ 8.4What to measure before there is anything to measure#

§The stage this whitepaper is written for often has no product in market. There is still something to measure, and it is the evidence discipline of Product Sprint: how many strangers completed the job in a prototype session, how many made a commitment at the end of a conversation, how many of the assumptions in the written position are validated, open or discarded. Those are the pre-launch cohorts. They are small, they are honest, and they are the numbers that should set the first goals.

§Vince Law offers a top-down way to choose the number rather than collect it, which fits this chapter's rule exactly. His GAME sequence runs goals, actions, metrics, evaluation: name the user goal and the business goal first, list the actions inside the product that serve them, only then turn those actions into a measure, and finally test the measure against real data and iterate. The order matters because it forces the metric to descend from a goal, so a number that traces to no goal is caught before it reaches a dashboard. That is the same discipline the strategy ladder uses for its own goals: every metric has a parent above it, or it is measuring for its own sake.

§The next chapter is the one metrics idea that changed in the last decade, and that this book asks a startup to know about before it needs it: the difference between a funnel and a loop.