Build Measure Learn vs. Learn Measure Build

Kent Beck's Itchy Goat metaphor for Lean StartupThe first time I saw Kent Beck speak he was going on and on about an itchy goat. I had no idea who he was or why he was talking about this goat or why he was scratching it on the back. But he said something that struck me and keeps coming back.

The Build Measure Learn loop is backwards.

There is a presumption there that if you start building something and slap some analytics on it then you will inevitably learn something.

[UPDATE: Reversing the BML loop is in fact noted in Eric Ries’ book The Lean Startup. I had just always recalled it from Kent’s talk.]

Step 1 Step 3 – Build a Minimum Viable Product

Building an MVP shouldn’t be about minimum effort or just taking out features. A Minimum Viable Product must be designed for validated learning.

Sure, if you just want to hack something fun together over the weekend, go for it. Release it, see what happens, decide if you want to continue. That’s certainly one way to approach things. (This is affectionately called the “Field of Dreams” approach by some.)

But if your goal is validated learning, the first thing you need to do is decide what to learn. Scientists don’t just start throwing chemicals into a vat and randomly feed it to babies to see what happens. They didn’t build the Large Hadron Collider for the hell of it.

Building random toys to find Product/Market Fit is like paying $40k a year to go to college and throwing darts at a course catalogue to pick your classes.

A good scientist forms a hypothesis, then carefully designs an experiment with a control group to measure the effect of that experiment. Only then do they implement the experiment and learn something.

In that regards, Minimum Viable Product is a terrible name. I prefer to build a Minimum Viable Test.

(Parker Thompson also proposed the term Minimum Viable Interaction which I also like tremendously.)

Step 2 – Measure What?

Let’s just throw it out there and see if it works!

Awesome! Do it! How will you know if it works?

Let me guess…you’ll slap google analytics on it, throw up a post on Hacker News, it’ll be voted to the top it meritocratic fashion, and then you’ll be able to tell if people sign up.

Here’s the problem:

  1. Getting to #1 on Hacker News? – A vanity metric.
  2. # of sign ups? – A vanity metric.
Your goal must be validated learning about Product / Market fit.

Product / Market fit does not mean, “does anyone want this thing I built?”

Just knowing that someone, somewhere out in the world want something you build doesn’t help you.

If there are enough dubiously curious individuals out there who want to buy the shake weight, then I guarantee you that no matter how idiotic your startup idea is, there is at least one individual who is willing to pay for it.

Who is that person? Why do they want it? How are you going to find that person? How many of them are there? Are they willing to pay for it?

At the cruz of the biscuit, Product / Market fit asks, “does this specific customer segment desire this specific value proposition?” So at the minimum you need a to measure what % of your target market sign up with a reasonable sample size.

That implies you either restrict your marketing to the customer segment, or you screen them out of your conversion somehow.

In other words, if you’re building a product to target soccer moms, then a viral posting on Hacker News has no bearing on your product, even if you have a 100% conversion rate. (If anything, that would imply that you are targeting the wrong customer segment.)

You can not measure your signup conversion rate without having a clear target market.

(BTW: “Everyone” is not a clear target market. More on that in another post.)

Step 3 Step 1 – Learn

Build Measure Learn vs. Learn Measure Build

So if you want to start with Build, please do so. I enjoy building random toys myself from time to time. But don’t kid yourself about what you’re learning.

If you really want to learn about your business, start by figuring out what you want to learn.

  1. Establish a hypothesis
  2. Determine a quantitative or qualitative method to evaluate that hypothesis.
  3. Build an experiment to test that hypothesis

So…what should I post next? Tweet to tell me what to write:

Show me how to test product market fit!

or

How can I do lean startup in my friggin' huge company?

Discussion (14 comments)

  1. Scott Lewis says:
    23.02.2012.

    Nice one.

    Couldn’t agree more.

    Are you a reverse engineer in your spare time? 😉

    Thanks for writing!

  2. Micah Alcorn says:
    23.02.2012.

    Very well said. I am experiencing this first hand. I built a product to scratch my own ‘goat’, and I am trying to learn about usage passively. My message has not been clear enough for my early testers to respond accordingly, and I haven’t been prepared to focus on one specific hypothesis.

  3. John McFarlane says:
    11.03.2012.

    Im glad that you pointed out # of sign ups? – A vanity metric.

    Im suprised that so many use this approach and either have a very vague description on a landing page with a mailing list subscription form or no info at all.

    Its almost funny how many have the typical “we’ll let you know when were ready” or something like that, but they haven’t considered whether people are ready for what they are building.

    Often the time between their email collection and anything being ready to test is so long that when i get an email letting me know that i can create an account that i might still remember signing up for it but cant for the life of me remember what first interested me.

    Often the anti-climax too of not having a standard login option and forcing the facebook login on you.

    1. Tristan says:
      13.03.2012.

      Totally concur.

      It’s very hard to go from a smoke test to real code and we all tend to get locked into a massive build after MVP. There needs to be an MVP 1.1

  4. Enny O says:
    10.05.2012.

    Hi Tristan,

    I could boast to be an expert and know all but am not. Thanks for this post. However, I’d like to ask between learning about one’s product and testing in the implementation of the Lean Startup, at what point should I be running to register my company name or betterstill, the patent and copyright?

    If I had to be practically lean in my works, I wouldn’t want to be running to the register a new business name everytime the former wasn’t viable.

    What are your thoughts please?

    1. Ray says:
      23.06.2012.

      Enny, don’t do it until you have to (validations); IMHO not until you people start paying you for your produce, ie when you need a bank account

    2. Tristan Kromer (@TriKro) says:
      23.06.2012.

      I agree with Ray but would temper that by asking, “how much risk is there?”

      If you’re got unlimited funds, it is not risky at all to spend money on trademarks, domain names, lawyers for patents, etc.

      In the case of some deep technological invention, a patent might be valuable down the road (or as social capital to be able to say, “Holder of 5 patents”) even if the business you were trying to execute on top of it was worthless.

      Personally, I am quick to buy domain names and slow to register trademarks. Copyrights are cheap and I used to do this immediately on writing a new song, but for a website, just write (c) copyright on the bottom. I never bother to register a company name, but I keep one LLC and do all my new business experiments and consulting under that LLC for liability protection.

  5. Hashim Warren says:
    12.09.2012.

    Where is Build-Measure-Learn reversed in Eric Ries’ book?

    1. Tristan says:
      12.09.2012.

      p. 78 and again on 201-202. Maybe some other places but I only just looked quickly in the index.

  6. Pingback: Survey Results: Are Developers More Productive In Scala? | Evolvable Me

  7. Salman Merchant says:
    14.03.2013.

    Doesn’t some small portion of your testing need to be with other customer segments, so you can see if some other part of a population besides your targeted customer segment is interested in your value proposition? Especially if you’re ONLY targeting your hypothesized target customer?

    1. Tristan Kromer (@TriKro) says:
      14.03.2013.

      I prefer to test segments in serial rather than parallel. If you test at the early stages in parallel you risk not having sufficient sample size to make an accurate judgement on any segment.

      In addition, it tends to fracture team focus.

      Better to stack rank the segments you’d like to focus on and then systematically test one after another.

      If the first one works, that doesn’t mean you can’t test an alternative segment to see if it’s even better.

  8. Evan L says:
    09.05.2013.

    I think this is a great wakeup call for entrepreneurs who misinterpret the lean startup methodology. However, I don’t agree that the cycle should be reversed.

    I have learned that the problem lies with the scale on which we perform the cycle. In fact, you outline the first full cycle in step 1 (learn): “build an experiment to test the hypothesis.” People tend to run away with the “build” part of the cycle instead of iterating at this scale (they fire cannon balls instead of bullets).

    From another angle: If your startup is simple enough to jump into learning before an experiment is performed, think bigger! This world has enough mobile payment systems, crowd funding platforms, and social media sites for pets as it is.

    1. Tristan says:
      09.05.2013.

      Speed is a great asset, but great speed in the wrong direction doesn’t get you to your goal.

      Putting learn first doesn’t mean “learn before doing”. It means figure out what you want to learn.

      In other words, steer in the right direction before you put your foot on the gas.

    2. Dan says:
      19.08.2013.

      I agree, I think there’s room for both. Whenever implementing a test in your product, it’s important to be clear about what results you expect and to make sure you can accurately collect data to compute your metrics. That said, there are times when you implement a test and you don’t get the expected results, but if you’re measuring everything you can notice unintended effects – very much like serendipity in scientific research (penicillin, teflon, viagra, etc).

      Tristan, if you’re doing rapid iteration right, if you’re moving in the wrong direction you can correct your path very early, and you won’t be moving in the wrong direction very long.

  9. Pingback: Procto » Articles » Minimum-Viable-Product Demystified (Part 2)

  10. Pingback: It’s MAD to Have a Separate Discovery Phase | Agile Product Discovery

  11. Pingback: It's MAD to Have a Separate Discovery Phase

  12. Pingback: The Ultimate Guide to Running a Lean Startup by @TriKro

  13. Pingback: The Rudder Fallacy – Adopting Lean Startup Principles | Digital Marketing Blog

  14. Pingback: It's MAD to Have a Separate Discovery Phase - Aaron Sanders | Agile Coach

  15. Pingback: The Rudder Fallacy – Adopting Lean Startup Principles | Cornel Lazar

  16. Pingback: Creating a culture of learning at The Guardian: our Product and UX Toolkit – Charlotte Gauthier

  17. Pingback: Lean validation: past, present and future

  18. Pingback: Iteration = Time to Learn, Not Time to Build

Got something to say?