15. Prioritizing experiments

§The question: of all the experiments we could run, which do we run next, and how do we run more of them?

§By now the possible experiments outnumber the weeks to run them. This chapter is how to choose, and then how to raise the number the team can run, because the rate of honest learning is itself the thing that compounds.

§ 15.1Impact, confidence, ease#

§Sean Ellis's ICE framework scores each candidate experiment on three: impact, how much it would move the target metric if it worked; confidence, how sure you are it will; ease, how little effort to run it. Score each one to ten, average or sum, rank, run the top.

§RICE, from Intercom, adds reach and is worth using when experiments touch different-sized populations. But at a startup, ICE's simplicity is usually right, and its real value is not the arithmetic, which is rough, but the conversation: scoring forces the team to say why an experiment matters and how sure it is, which surfaces the easy-but-pointless experiments that would otherwise fill the backlog.

§ 15.2Aim at the weakest stage#

§Scoring happens within a target, and the target is the funnel's weakest stage from chapter 10. An experiment that would brilliantly improve a stage that is already fine scores high on impact and is still the wrong experiment, because the leak is elsewhere. First find the stage where the most is lost; then prioritize experiments on that stage. This is the discipline that keeps ICE from optimizing a corner of a business that is bleeding somewhere else.

§ 15.3Experiment velocity#

§The deeper metric is not any single experiment's score. It is experiment velocity: how many honest loops the team completes per unit of time. Sean Ellis built a whole growth process around cadence for this reason, and Reforge treats velocity as a first-class growth metric.

§The logic is the logic of the whole whitepaper. Each loop produces learning; learning compounds; so the rate of learning is the rate at which the whole system improves. A team running four honest experiments a month learns twelve times as much in a quarter as a team running one, and the gap widens, because the faster team's earlier learnings make its later experiments sharper.

§The word that keeps velocity honest is "honest." Velocity is not running more badly-designed tests. It is completing more full loops: a real hypothesis, a real threshold set in advance, a real read, a real decision. A team that games velocity by running underpowered tests on trivia has high velocity and zero learning, which is worse than low velocity, because it feels productive.

§ 15.4Removing the constraints on velocity#

§Raising velocity means removing whatever caps it, and the caps are usually not ideas. They are instrumentation that takes days to answer a question, a review cadence that batches decisions, or a build step for experiments that should have been fake doors. Chapter 16 and 17 are about exactly these constraints. A team that wants to learn faster fixes its instrumentation and its cadence before it demands more ideas.

§ 15.5What you leave with#

§A backlog scored by impact, confidence and ease, filtered to the funnel's weakest stage. The next experiments chosen from the top. And experiment velocity tracked as a metric of the system itself, defined as complete honest loops, with the current constraints on it named for Part IV to fix. Part IV is the machine that keeps the loops turning.