How do you turn experiment results into insight?

The output of experimentation is not a series of shipped changes. It is a growing understanding of the users, and that understanding feeds back into strategy and the business model. Each experiment result should be turned into an insight, a durable statement about the users, and the insights should feed the fourth and fifth whitepapers, so that direction and economics stay tied to what behavior has proven.

How does experimentation feed back into strategy?

The output of experimentation is not a series of shipped changes. It is a growing understanding of the users, and that understanding feeds back into strategy and the business model. Each experiment result should be turned into an insight, a durable statement about the users, and the insights should feed the fourth and fifth whitepapers, so that direction and economics stay tied to what behavior has proven.

How does learning compound in a startup?

The output of experimentation is not a series of shipped changes. It is a growing understanding of the users, and that understanding feeds back into strategy and the business model. Each experiment result should be turned into an insight, a durable statement about the users, and the insights should feed the fourth and fifth whitepapers, so that direction and economics stay tied to what behavior has proven.

Learning that compounds · Product Experimentation

§The question: what do all these experiments add up to, and where does the understanding go?

§An experiment ships a change and produces a number. That is the visible output and the least important one. The real output is a piece of understanding about the users, and this chapter is about capturing it and feeding it back into the direction and the economics, so that the collection's closes.

§ 18.1From result to insight#

§A result is "the two-step onboarding raised day-one activation from 40% to 58%." An is "our users abandon on effort, not on confusion; every step we remove helps and every explanation we add hurts." The result is about one experiment. The insight is about the users, and it applies to the next ten experiments, the next feature, the next onboarding.

§Turning results into insights is the act that makes learning compound. Gary Klein's research on how insights form points at the mechanism: an insight is usually a pattern noticed across several observations, or a contradiction that forces a new explanation. The weekly review is where results accumulate; the insight is the periodic step back to ask what the results, together, are saying about who these users are.

§ 18.2Feeding strategy#

§Eric Ries's validated learning is the unit of progress precisely because it feeds back. The insights from experimentation are the fourth whitepaper's ladder was built on assumptions, and now some of those assumptions have behavior behind them. A bet whose critical question has been answered by a retention curve is no longer a bet; it is a fact, and the ladder should be updated. A goal that has been hit or missed changes the sequence of what comes next.

§This is the loop the whole collection describes, closing. Discovery produced a position. Strategy turned it into a ladder of bets. Business design attached economics. Experimentation tests the bets and the economics against behavior, and the results flow back up: confirmed bets become foundations, refuted bets trigger the iterate-or-pivot decision, and the ladder is rebuilt on firmer ground each cycle.

§ 18.3Feeding the model#

§The insights feed the fifth whitepaper too. The were built on assumptions, cost to serve, retention, willingness to pay, and experimentation replaces those assumptions with measured numbers. The unit economics sheet is not written once; it is updated each time an experiment produces a real number for one of its lines, and the "what has to be true" list is checked off, row by row, as behavior confirms or refutes each hypothesis.

§A team a year in should be able to open its business design list and see most rows now carrying a real number and a date confirmed, with the few still-open rows as the current experimental focus. That is what it looks like when learning compounds into a business.

§ 18.4What you leave with#

§For each experiment, an insight, a durable statement about the users, not just a result about the change. A periodic step back to find the patterns across insights. An updated strategy ladder, with confirmed bets marked as foundations and refuted ones triggering a decision. And an updated business design list, with rows now carrying measured numbers. The collection's loop is closed: behavior feeds direction, and direction feeds the next behavior. Part V is how the whole thing fails.

18. Learning that compounds