10. The ladder of fidelity
Each rung of the ladder answers a different question at a different cost. A smoke test asks whether anyone wants it. A concierge asks whether you can deliver the value by hand. A Wizard of Oz asks whether the experience works before the machine exists. A navigable prototype asks whether a stranger can use it. Pick the rung by the question, not by how impressive it looks. The rule is the cheapest thing that can be wrong.
- What is the difference between a smoke test, a concierge MVP and a Wizard of Oz test?
- How do you choose what kind of prototype to build?
- What is the cheapest way to test a startup idea?
§The question this chapter answers: what is the cheapest thing we could build that would be wrong if we are wrong?
§The word prototype hides four different objects, and teams that do not separate them build the expensive one to answer the question the cheap one would have answered. This chapter is . The sixth whitepaper owns everything that happens after launch; this one stops at the first stranger.
§ 10.1Rung one: the smoke test#
§The question: does anyone want this enough to act? The object: a landing page, an ad, an email, a post. A description of the thing and a way to say yes: join a list, book a call, pre-order. What it can tell you: whether the one sentence from chapter 9 makes a stranger move. Conversion from view to action, and, more usefully, the words people use when they reply. What it cannot tell you: whether they will use it, pay for it, or keep it. A sign-up is a compliment with an email address attached.
§Buffer's founder published the canonical version in 2011: a two-page site, a pricing page that did not yet have a behind it, and a note when someone clicked. Alberto Savoia calls this pretotyping, testing the it before the how. The 2020s version is cheaper still, and the failure mode is the same as the community pitch in chapter 4: the page promises the thing the market has been asking for, forty people sign up, and the founder builds the wrong product with great confidence.
§ 10.2Rung two: the concierge#
§The question: can we deliver the value by hand, and does the customer value it when we do? The object: you, doing the job manually for a handful of customers, in the open. No software, or a spreadsheet. What it can tell you: whether the value is real, what delivering it actually involves, what customers do with it, and what they will pay. This is the richest source of on the ladder, because you are inside the customer's operation. What it cannot tell you: whether it scales, or whether the experience of a product would land the way the experience of a person did.
§Eric Ries gave it the name. It is the strongest option for business software, where the customer cares about the outcome and not the interface, and it has a second benefit: after three months of doing the job by hand you know the requirements better than any interview could tell you.
§ 10.3Rung three: the Wizard of Oz#
§The question: does the experience work, before the machine that powers it exists? The object: a real-looking interface with a person behind it. The customer types; a human answers. The customer uploads; a human . What it can tell you: how people behave with the product as a product. What they ask it, how they phrase things, where they hesitate, what they expect to happen next. What it cannot tell you: whether the machine can do what the human did.
§In the AI era this rung has inverted in an interesting way. For twenty years the wizard was a human pretending to be software. Now the wizard is often a general-purpose model with a long prompt, pretending to be a specialized product, and the question is whether the specialized product is worth building at all or whether the prompt is the product.
§ 10.4Rung four: the navigable prototype#
§The question: can a stranger use it? The object: screens a person can tap through, on their own device, with realistic data, and nothing behind them. What it can tell you: where people get stuck, what they expect, what they think a thing is for, what they ignore. Whether the loyalty unit makes sense. Whether the check-in gesture is obvious. Everything in chapters 11 and 12. What it cannot tell you: whether they want it. A stranger can navigate a prototype flawlessly and never open the product.
§This is the rung AI changed most. Two working days from a written brief, by people who have never designed a screen. Because it is now cheap, it is now overused: teams build it to answer questions that a smoke test or a conversation would have answered, and the prototype flatters them into skipping both.
§ 10.5The cheapest thing that can be wrong#
§The rule that picks the rung: build the cheapest thing that would be visibly wrong if your assumption is wrong. Not the cheapest thing, which is nothing. Not the most convincing thing, which is the finished product. The cheapest thing whose failure you could not explain away.
§A smoke test with a conversion rate can be wrong. A prototype in a stranger's hands can be wrong. A demo to your advisors cannot be wrong, because they will be kind, and so it is not a test.
§ 10.6What you leave with#
§For each surviving canvas, the rung you chose and the question it answers, in one line. The assumption that would be visibly wrong if it fails, and what wrong would look like in numbers or in behavior. If the rung is a smoke test or a concierge, you can start today. If it is a prototype, the next chapter is how to write the brief.
Build the cheapest thing that can be wrong. If it cannot be wrong, it is not a test.
- Alberto Savoia, The Right It (2019), on pretotyping. www.pretotyping.org
- Eric Ries, The Lean Startup (2011), on concierge and Wizard of Oz MVPs. theleanstartup.com
- Joel Gascoigne, Idea to Paying Customers in 7 Weeks: How We Did It, Buffer (2011). buffer.com/resources/idea-to-paying-customers-in-7-weeks-how-we-did-it
- OpenAI, Introducing ChatGPT, released as a research preview (November 2022). openai.com/index/chatgpt