17. Instrumentation

§The question: what do we need to measure so our experiments are readable, and where do we stop?

§Instrumentation is where experimentation either becomes possible or becomes theater. Too little and every experiment is unreadable; too much and the signal drowns in dashboards. This chapter is about measuring exactly what the loops need, which is less than teams fear and more than they usually have where it counts.

§ 17.1Measure what the experiments need#

§The principle inverts the usual instinct. Do not instrument everything and hope the answers are in there somewhere. Start from the experiments you are running and the funnel you are diagnosing, and instrument what those require. Alistair Croll and Benjamin Yoskovitz's rule holds: a good metric is one that changes a decision, and an event you would never act on is an event not worth capturing.

§For a small team that means, roughly: the funnel stages as events, so you can see where users leave; user tagging by cohort and by acquisition channel, so you can read retention and channel quality; and the specific events that define activation and the core retained action. That is a short list, and it is enough to run every experiment in this whitepaper. Everything beyond it is added when a specific experiment needs it, not before.

§ 17.2The two failures#

§Under-instrumentation makes experiments unreadable. The team runs an onboarding experiment but never instrumented the onboarding steps, so it knows activation changed but not which step drove it. It knows retention differs by channel but never tagged the channel. Every experiment produces a number and no insight, because the funnel underneath it is invisible.

§Over-instrumentation buries the signal. The team captures hundreds of events, builds thirty dashboards, and nobody can find the one number that matters, because it is next to two hundred that do not. Amplitude's North Star playbook exists partly to fight this: pick the one metric that matters and the few inputs that drive it, and let the rest be queryable rather than dashboarded. A dashboard is a claim that a number deserves daily attention, and most numbers do not.

§ 17.3Data quality is the hidden foundation#

§Kohavi's book spends more pages on data quality than on statistics, and the reason applies to the smallest team: an experiment read off bad data is worse than no experiment, because it is confidently wrong. The events must fire reliably, the cohorts must be defined consistently, and the same user must not be counted twice or dropped. A team that has not checked that its activation event actually fires when it should is measuring a definition, not a behavior.

§The small-team version of data quality is verification by hand: after instrumenting an event, do the action yourself and confirm the event appears, correctly attributed, once. It is tedious and it is the difference between a system that learns and a system that hallucinates.

§ 17.4Maintainable by the team that builds#

§The instrumentation must be maintainable by people who also ship the product. That argues for a small, standard set of tools rather than a bespoke pipeline, for events named by a convention the team agreed, and for a shared definition of each metric so that activation means the same thing in every conversation. The heaviest analytics stack is worthless if maintaining it competes with building the product; the lightest one that answers the questions is the right one.

§ 17.5What you leave with#

§A short instrumentation set: funnel stages as events, users tagged by cohort and channel, the activation and core-retention events defined precisely and verified by hand. One metric named as the current focus with its few inputs, and everything else queryable rather than dashboarded. And a shared, written definition of each key metric, so the team argues about results, not about what the words mean. The next chapter is how the results become insight that feeds the rest of the collection.