How long should you run a product experiment?

Run an experiment long enough to learn and no longer. At a startup the numbers are small, so honest statistics matter more, not less: a difference of two out of twenty is not a finding, and calling it one is the most common self-deception in early experimentation. Some questions cannot be answered with A/B tests at your scale, and for those the honest tool is a qualitative read of a few real users, clearly labeled as such.

How many users do you need for a valid A/B test?

Run an experiment long enough to learn and no longer. At a startup the numbers are small, so honest statistics matter more, not less: a difference of two out of twenty is not a finding, and calling it one is the most common self-deception in early experimentation. Some questions cannot be answered with A/B tests at your scale, and for those the honest tool is a qualitative read of a few real users, clearly labeled as such.

How do you learn from experiments with small sample sizes?

Run an experiment long enough to learn and no longer. At a startup the numbers are small, so honest statistics matter more, not less: a difference of two out of twenty is not a finding, and calling it one is the most common self-deception in early experimentation. Some questions cannot be answered with A/B tests at your scale, and for those the honest tool is a qualitative read of a few real users, clearly labeled as such.

Duration and sample · Product Experimentation

§The question: how long do we run it, how many users do we need, and when is a difference real rather than a coin flip?

§This chapter is where enthusiasm meets arithmetic. A small team wants a fast answer, and the honest answer is often that the sample is too small to give one, and knowing that is worth more than a confident number that is noise.

§ 4.1Long enough, no longer#

§An experiment runs until it has enough data to cross the threshold with confidence, or enough to say it will not. Two forces set the length. Below, the sample has to be large enough that the difference you care about is distinguishable from chance. Above, the has to be short enough that the team can turn it many times before the runway ends.

§The second force tempts teams to stop early, and Evan Miller's warning is the one to internalize: peeking at an experiment and stopping when it looks good inflates false positives dramatically. The stopping point is decided in advance, from the sample size the effect requires, not from watching the number until it says what you want.

§ 4.2The small-numbers problem#

§At a startup, the sample is small, and small samples lie. A feature shown to twenty users that eight complete versus six in the control is a difference of two people, and two people is well within what a coin flip produces. Presented as "completion rose from 30% to 40%," it sounds like a finding. It is a coin landing heads twice.

§The arithmetic is unforgiving and worth knowing roughly: to detect a ten-point difference in a rate with any confidence, you need hundreds of users per arm, not dozens. Most early-stage A/B tests do not have the sample to detect the effects they claim, which means most early-stage A/B tests are theater. Kohavi's book is explicit that underpowered tests are worse than no tests, because they produce confident wrong answers.

§ 4.3When you cannot A/B, read qualitatively#

§Many of the most important early questions cannot be answered by a controlled test at your scale, and the honest response is not a badly powered A/B test. It is a qualitative read of a small number of real users, labeled clearly as what it is.

§Rahul Vohra's Superhuman engine is the model. With too few users for statistics, the team asked one survey question, segmented the answers, and read the open-text responses of the very-disappointed users by hand. That is not an A/B test and does not pretend to be. It is a structured qualitative read that produced a decision, and its honesty is in never claiming a p-value it did not have.

§The rule: if the sample can support statistics, use them properly. If it cannot, use a qualitative read of a few real users and say so. What you must never do is run an underpowered A/B test and report its difference as if it were significant.

§ 4.4Cohorts over snapshots#

§At every scale, the honest view is the : this group of users who started in this week, followed over time. A snapshot, "40% of users are active", mixes people who joined yesterday with people who joined a year ago and tells you nothing about whether the product is getting better. The next chapters lean on cohorts constantly, and the reason is here: a cohort is the smallest honest unit of after-launch learning.

§ 4.5What you leave with#

§For each experiment: the sample size the effect requires, computed roughly before the run, and an honest verdict on whether you can reach it. The fixed stopping point. And, when the sample is too small, a deliberate choice to read qualitatively instead, labeled as a qualitative read and not a test. Part II applies all of this to the first question after launch: does the thing work?

4. Duration and sample