There are two lean startup methods that are often confused with one another. As a result, they’re often misused: Concierge test and Wizard of Oz test
tl;dr: Wizard of Oz is for testing a specific solution hypothesis, Concierge is for generating ideas. Use each for the right reasons. You can read more by downloading the Real Startup Book which is being published in two week iterations.
Wizard of Oz Test
Aardvark connected people with questions to people with expertise. Cardmunch managed to transcribe blurry photos of business cards better than any other Optical Character Recognition (OCR) system at the time. Neither had any technology or algorithm behind them.
The technology was human beings.
Cardmunch leveraged Amazon’s Mechanical Turk while Aardvark used interns. Instead of building complex algorithms both just faked it. Since neither needed a real time response to provide their core value propositions, there was no need to build anything.
To the user, the products delivered exactly what they promised. A little slow perhaps…but the value was there.
(Note: Lean Startup did not invent this technique. It was already being used in the lab by 1975 by J. F. Kelley.)
Concierge testing was used by companies such as Wealthfront and Food on the Table. The approach is similar in that there is no technology involved. Both companies performed the service they were offering as a concierge at a hotel might. Purely manually.
Wealthfront would work with its customers to provide wealth management advice and portfolio management. They would sit with them, pen and paper in hand, and figure out solutions to wealth management issues step by step.
Manuel Russo used to personally go shopping with his customers to help them plan their meals. (Not very scalable!)
So both Wizard of Oz and Concierge testing provide the value proposition manually without any technology. What’s the difference?
The difference is that in one case, the user knows about the concierge, while the Wizard of Oz is hidden from the user. This is a key difference for the user experience.
In a Wizard of Oz test, the user experience is a complete simulation of the actual intended product, but with a slight time delay since a human cannot react as fast as a computer. The delivered value proposition is almost identical but slightly under performing.
In a Concierge test, the value proposition is actually significantly higher than the eventual product. That’s because the customer is aware and is interacting with a real human being.
Humans are Social Creatures
There are some circumstances where we do not want to interact with a human being. For me, I don’t want to talk to anyone at Macy’s. Malls and department stores just kind of freak me out in general.
For everyone else, just imagine that you just typed a search into WebMD to do a little private research and all of a sudden a window pops up with a live human being on a webcam to tell you all about your search query for genital herpes. Fun!
Compare that to Food on the Table where we’d get to go shopping with Manuel Russo. Normally, I find shopping a bit of a chore, but Manuel is a pretty fun guy. I was delighted to meet him a few years ago in Austin and I would be equally delighted to go shopping with him. (Manuel seems to know all the best places to get food in the city and is hysterically funny.)
A Concierge test has an extra value proposition that a Wizard of Oz test does not. Or at the very least, the test is severely affected by the visible presence of a human being.
Generative vs. Evaluative
This brings us back to the distinction of Generative Research vs. Evaluative Experiments.
Wizard of Oz testing allow us to test a clearly defined hypothesis about what solution will provide the value proposition to the customer. We test it by simulating the user experience as precisely as possible without technology. The lack of technology means we can rapidly test a variety of hypothesis and rapidly iterate.
Since Wizard of Oz allows us to test and falsify a solution hypothesis, it’s an Evaluative Product Experiment.
Concierge testing has a huge human factor which biases the test. If we attempt to use a Concierge test to validate our solution hypothesis, we’re liable to get a bias towards a false positive or perhaps even to a false negative in some circumstances. It’s simply the wrong type of test to falsify a clear hypothesis.
Concierge testing is very bad when we have a clear idea of the solution, but very good when we’re not exactly sure what the solution should be!
There were many unknowns including how to best represent the portfolio information to the customer in an intelligible way to them to make their own decisions.
Concierge testing allows us to work with the customer without having a clear solution hypothesis. A vague idea, or even no idea, will sometimes be enough to muddle through the process while talking with the customer and getting constant feedback.
We might think that it’s impossible to go to a customer and just sell the promise of a solution without having any clear idea of how to provide it. It’s not.Selling solutions that don't exist and then winging it is common place. It's called Consulting. Click To Tweet
(Note: Lean startup didn’t come up with Concierge testing either. Consulting has been around since the before the pharaohs.)
By the time we’ve manually performed the task a few times for our consulting clients, we have a much clearer idea of what form an automated solution can take and what the minimum viable feature set looks like.
The solution is not yet validated, but we can take that hypothesis and begin the process of evaluating experimentation with a Wizard of Oz test, paper prototyping, usability tests, etc.
Lessons LearnedWizard of Oz hides the human. Concierge flaunts the human. Click To Tweet The Wizard of Oz method is for evaluating a solution hypothesis. Click To Tweet The Concierge method is for generating solutions ideas and hypotheses. Click To Tweet
Confusing them can lead to faulty reasoning and flawed business models.
For more ways to test your solution or figure out your minimum viable feature set, download the Real Startup Book which is being co-created by 40+ lean startup enthusiasts: