False Positives and Product / Market Fit

(This is part of a series of posts about testing product/market fit. You can download the Product/Market Fit Storyboard here. If you’re in San Francisco, you can also join the Your MVP Sucks! Workshop.)

As I’ve mentioned, being a lean startup is hard. Beyond the psychological pressures of trying desperately to fail at a rapid rate, there are a million ways to test a hypothesis, and a million hypotheses.

How do we know which type of test to use, and when to use it?

  • Should we do a smoke test?
  • A concierge test?
  • Talk to our customers?

All we want is to build something people want. Which one of these tests will actually tell us if we have the fabled Product / Market Fit?

product market fit 1024x554 False Positives and Product / Market Fit

(This is a giant topic that I’m going to explore over the next several posts. You can cut to the chase by downloading the Product / Market Fit Storyboard here. If you’re in San Francisco, you can also join the Your MVP Sucks! Workshop.)

Let’s start with the obvious…

What is Product / Market Fit?

Product/Market Fit was a phrase coined by Marc Andreessen. As Kali Albright notes in his answer on Quora, Marc said,

Product/market fit means being in a good market with a product that can satisfy that market.

For context, Marc is a venture capitalist. So a “good market” is one that is sufficiently large and growing at a rate to warrant a VC being even remotely interested.

Scaling Marketing

startup marketing pyramid 300x289 False Positives and Product / Market FitSean Ellis, who is often credited with popularizing the term, had a similar context. He said that he was only able to effectively scale marketing after the company had achieved PMF and he used his Customer Development Survey survey to measure their readiness. Product/Market Fit was the foundation of his Startup Marketing Pyramid.

So, Product/Market Fit is about scaling.

Put simply, it’s when you stop wondering if your product actually solves a problem and you start wondering how you’re going to deal with all the people trying to buy your product and how to get even more of them to your storefront.

How do I know if I have Product / Market Fit?

Sean used his Customer Development Survey to measure Product/Market Fit. He specifically said that a 40% or higher result was necessary to have Product / Market Fit. The survey is:

How would you feel if you could no longer use [product]?

  1. Very disappointed
  2. Somewhat disappointed
  3. Not disappointed (it isn’t really that useful)
  4. N/A – I no longer use [product]
very disappointed customer 277x300 False Positives and Product / Market Fit

A Very Disappointed Cusomter

He went on to say:

If you find that over 40% of your users are saying that they would be “very disappointed” without your product, there is a great chance you can build sustainable, scalable customer acquisition growth on this “must have” product… Those that were above 40% are generally able to sustainably scale the businesses; those significantly below 40% always seem to struggle.

Asked specifically about this at the March 2010 Lean Startup Circle meetup in San Francisco, Sean repeated that the >40% result was necessary but not sufficient and I know the failings of a misinterpreted survey first hand.

StartupSquare

 False Positives and Product / Market FitMy first startup in Silicon Valley, StartupSquare, had >40% on the Customer Development Survey and unequivocally did not have PMF. We had Problem / Solution Fit, but not Product / Market Fit.

Our product simply didn’t work that well. In fact, it didn’t do too much of anything.

Instead, we had measured our customer’s great burning desire for a solution, but had not succeeded in creating that solution.

Furthermore, the market was too small at that time to merit additional investment in creating a better solution. Any additional time or money spent would have been wasted on a fundamentally broken business model.

It failed both Marc Andreessen and Sean Ellis’ definitions.

Jargon

So here is my more specific and overly wordy definition:

Product/Market Fit is sufficient demand in a clearly defined marketplace for a product delivering a clearly defined value proposition to allow efficient (human or financial) capital expenditure to scale value creation.

If scaling the company results in increased costs without additional returns (either in revenue or, at the very least, company valuation) then the company is prematurely scaling prior to Product/Market Fit.

False Positives

customer development survey false positive 199x300 False Positives and Product / Market FitI suspect survey.io can give a false positive result when the product is early stage enough that customers are answering based on the promise of a solution instead of the actual product.

(I admittedly have not, and will not, do the more rigorous academic research to prove this conclusively. But I only need one example to show that a false positive is possible.)

>40% “very disappointed” on survey.io is necessary but not sufficient to have Product/Market Fit

So while having <40% on Sean’s survey probably does indicate that you do not have PMF, >40% is a very very strong indicator, but does not necessarily mean with certainty that you have product/market fit.

Product / Market Fit and Business Models

A single survey will never tell you if you have Product/Market Fit

Over the next few posts, I’m going to delve into this topic and how I use Product/Market Fit Storyboarding to create a roadmap of experiments to prove various hypotheses driving towards Product/Market Fit.

(If you’d like to cut to the chase, you can download the  Product / Market Fit Storyboard here. If you’re in San Francisco, you can also join the Your MVP Sucks! Workshop.)

(Illustration credit for the good ones goes to Emily Chiu)

So…what should I post next? Tweet to tell me what to write:

Show me how to test product market fit!

or

How can I do lean startup in my friggin' huge company?

11 comments

  1. Great article on False positives and P-M fit.
    I very much resonate with the Problem-Solution vs. Product-Market false positive. It was a false positive in that sense that distracted us and ultimately wasted too much of our resources at Selfless before we really understood how to step back and asses this was going on.

    I’m excited to see your next posts about storyboarding a roadmap of experiments.

    • Tristan says:

      Thanks Josh! I’m starting to think that there’s a third phase in there that we’re not talking about.

  2. Pingback: Самые важные ссылки для предпринимателя (6 выпуск) | Dimaslennikov.ru

  3. Pingback: The Four Parts of a Minimal Viable Product by @TriKro

  4. Hank says:

    When you just have a prototype, it is very hard for customers to say “Very disappointed” without your product, since they have live for so long without your product. So when would be the best time to run the survey? after product has been in the market for a while?

    • Tristan says:

      Hi Hank,

      I don’t think there is a one-size-fits-all answer.

      I’m more of a fan of measuring using other metrics such as retention. But if your product is a single use item like “remove your unwanted search results from google” or “buy this t-shirt”, maybe retention is the wrong metric.

      I personally would use Sean’s survey in addition to other metrics, not as a standalone solution for measuring product/market fit.

      That said, you should use the survey after you believe that you are delivering value to the customer.

      If you are really delivering unique value, the customer will be very disappointed to lose your service.

    • Hank,

      Sean talks about running the survey multiple times, even with the same users. You want to see the Very Disappointed percentage rising each time, and that means you’re getting closer to P/M fit.

      At Proof, we believe the Sean Ellis test is a really good test, but we don’t rely on it solely. Like Tristan said, track this and other metrics.

      • Tristan says:

        Dan Martel also use Sean’s survey for segmentation, which I thought was quite clever. Focus on turning the “somewhat disappointed” to “very.”

  5. Pingback: Customers and Causality by @TriKro

  6. Pingback: Selling a Kick in the Ass by @TriKro

  7. Pingback: Minimum Viable Product: Your ultimate guide to MVP (+great examples) | Fast Monkeys- official blog

Leave a Comment

Your email address will not be published. Required fields are marked *