Build Measure Learn vs. Learn Measure Build

Kent Beck's Itchy Goat metaphor for Lean Startup: build measure learn

In 2009 I saw Kent Beck speak for the first time at the very first Lean Startup Conference. Back then it was called the “Startup Lessons Learned” Conference. His topic? Build Measure Learn….and goats.

Kent was going on and on about an itchy goat. I had no idea who he was or why he was talking about this goat or why he was scratching it on the back. It was weird and confusing. Even Eric Ries looked a bit baffled standing on the sidelines. But it was brilliant. Kent said something that stuck with me.

The Build Measure Learn loop is backwards.

But let’s start with some background. (If you’re already very familiar with lean startup, you can skip the background.)

What is the Build Measure Learn Loop?

build measure learn loopThe Build-Measure-Learn (BML) loop is the most fundamental concept of the Lean Startup. It represents the basic process of rapid iteration based on direct consumer feedback on functional products. This rapid iteration validates or invalidates basic assumptions about the market need and whether the product satisfied that need.

The basic concept is similar to a number of loops found in other domains such as:

  • Think-Make-Check in Lean UX
  • Observe-Orient-Decide-Act (OODA) used in the US Air Force
  • Plan-Do-Check-Act (PDCA) in Quality Control
  • Assessment–Diagnosis–Planning–Implementation–Evaluation (ADPIE) in nursing

The BML loop isn’t really anything new. Every two year old is familiar with “If at first you don’t succeed, try, try again.”

What’s new about Build-Measure-Learn?

The most common differences emphasized by advocates of lean startup are:
  1. Speed
  2. Data
  3. Business focus

Speed

Lean startup emphasizes speed above all else.

Partly, that’s a result of being created in the technology sector by a software engineer. Silicon Valley is famous for emphasizing rapid iteration to get to market faster.

24 hour caffeine fueled hackathons are the norm. Participants are expected to showcase working software at the end. Prizes or just social approval, are the reward. Just watch Mark Zuckerberg (played by Jesse Eisenberg in The Social Network) to get the idea.

There are other industries where one can create a product in 24 hours. Personally, I’m a mediocre cook, but I could prototype a working restaurant menu in 24 hours. But there is no other industry where a product could be ready to scale to tens of thousands of users almost instantaneously.

Data

In lean startup, measuring has a strong emphasis on quantitative data.

The rapid scale of the tech industry and ability to measure detailed user behavior led to more data than anyone had ever had. The amount of data and detail available on a website is like Elf on the Shelf on overdrive. Every click, every mouse move is being tracked whether or not cookies are enabled.

That emphasis has mellowed over the years as lean startup has overlapped and merged with design thinking and user research techniques. But many lean startup evangelists still have a strong bias towards quantitative data above all else.

Business Focus

Lastly, lean startup is not about testing code or a product. That rapid iteration isn’t meant to determine if the product is working.

The Build Measure Learn loop is about testing a Business Model.

Lean Startup comes from agile software development and shares many of its principles. While agile evangelists will advocate many of the exact same tactics as lean startup, the primary focus is on building software.

Lean Startup explicitly focuses on testing a complete business model. That means simply producing working software is not a complete iteration. That working software needs to interact with customers. So the data doesn’t relate to error or defect rate, but relates to whether the business model is viable. (Hence the V in relative concept of Minimum Viable Product.)

Now back to Kent Beck…

Kent Beck’s Itchy Goat

The story Kent told involved baby goats, adult goats, and a chainsaw. I won’t get into but no goats were harmed during the story. It was a story about figuring out how to make his goat happy.

Personally, I have no idea how to tell if a goat is happy. To clarify, yes. Kent Beck owns goats. Yes, it was a strained metaphor. No, I’m not going to explain the story more than that. It was hard to capture and no one other than Kent Beck could have pulled it off. Don’t overthink it.

Bottom line is that through trial and error, he found the itchy spot. He suddenly received an “outsized reaction” for an activity he had been repeating in a number of different places to no effect.

But then Kent said, ”the Build Measure Learn loop is backwards.”

What’s wrong with the Build-Measure-Learn?

The BML loop presupposes that if we start building something and slap some analytics on it then we will inevitably learn something.

That means building a Minimum Viable Product.

Build Something

So we hack something fun together over the weekend, release it, see what happens. Then we can decide if we want to continue.

That’s certainly one approach. Let’s just throw it out there and see if it works!

How will we know if it works?

Measure Something

Slap google analytics on it and throw up a post on Hacker News or Reddit. The post will be voted to the top in meritocratic fashion, and then we’ll see if anyone sign up.

Here’s the problem:

  • Getting to #1 on Hacker News? – A vanity metric.
  • # of sign ups? – A vanity metric.

There are over 7 billion people on the planet. If someone is willing to buy the shake weight, someone will sign up for whatever idiotic startup idea we come up with. Even if it’s just as a joke.

But getting curiosity sign ups is insufficient.

Who is that person that signed up? How many sign ups do we need for success? Why do they want it? How many of them are there? Are they willing to pay for it?

Learn Nothing

If we’re building a product to target soccer moms, then a posting on Hacker News has no bearing on our product. If anything, it’s going to give us a false negative because we’re talking to the wrong people.

We can not measure our sign up conversion rate without having a clear target market.

(Note: “Everyone” is not a clear target market. More on that in another post.)

If Build Measure Learn is just trial and error, then it’s no better than hope. This is what is affectionately called the “Field of Dreams” approach by some and “Spaghetti testing” by others.

Build it, and they will come!

It’s the same thing, just faster. And that is not what Eric Ries was talking about in The Lean Startup.

Build Measure Learn Backwards

To learn quickly in any domain, we have to focus and have a clear plan of how and what we want to learn.

If we want to play the guitar, just picking it up and strumming isn’t going to get us very far very fast. To learn quickly, we need to break it down to simple elements and practice. Melody and harmony. Different chords and scales. Left hand and right hand.

There’s a clear goal right at the beginning of each practice session to focus on one specific element and master it.

Same goes for the Lean Startup. To learn quickly, we reverse the BML loop.Experiment and Research: Question - Data

Step 1 – Identify the Learning Goal

The first thing we need to do is decide what to learn. Scientists don’t just start throwing chemicals into a vat and randomly feed it to babies to see what happens. They didn’t build the Large Hadron Collider for the hell of it.

A good scientist forms a hypothesis, then carefully designs an experiment with a control group to measure the effect of that experiment. Only then do they implement the experiment and learn something.

So start the loop by figuring out what we’re trying to learn. (Our team at Kromatic calls this the Learning Goal.)

This goal is best phrased as a question. Don’t say, “I want to learn about product pricing.” Say, “What price will our customers pay for our product?”

Step 2 – Specify the Data

Rephrasing the learning goal as a question is just a cognitive trick. Thinking about pricing in the abstract seems hard. Especially if we’ve never done it before.

Framing our learning goal as a question makes it easier to figure out what sort of data would help us answer this question. We need to know what to measure.

Our question is, “What price will our customer pay for our product?” Based on that, there are some clear pieces of data that will help us:

  • % of people who actually pay $10, $20, or $9.99 for the product
  • The average price paid for a comparable product
  • The qualitative degree of the customer’s need for a solution

Each of these potential data points is something we can now go and get.

Step 3 – Select a Method

This is not where we go and build a Minimum Viable Product. We should build a Minimum Viable Test.

(Parker Thompson also proposed the term Minimum Viable Interaction which I also like tremendously.)

We know what data we need, so we should go out and get it with the least amount of effort possible.

To get the “% of people who actually pay $10, $20, or $9.99 for the product,” we don’t necessarily need to build a product.

We could try just building a landing page. Or we could even just attempt to sell the product face to face. We don’t even need the real product, a prototype or even a fancy box that looks like the real product would suffice.

To find “the average price paid for a comparable product” then we don’t need to build anything. We should just go shopping.

If we need “the qualitative degree of the customer’s need for a solution” then we should simply go talk to customers.

The qualitative degree of the customer’s need for a solution

Lessons Learned

So if we really want to start with Build, let’s do it. I personally enjoy building random toys from time to time just for the fun of it. But we shouldn’t ourselves about what we’re learning.

If we really want to learn about our business, we need to start by figuring out what we want to learn.

  1. Identify the Learning Goal
  2. Specify the Data
  3. Select a Method

Discussion (14 comments)

  1. Scott Lewis says:
    23.02.2012.

    Nice one.

    Couldn’t agree more.

    Are you a reverse engineer in your spare time? 😉

    Thanks for writing!

  2. Micah Alcorn says:
    23.02.2012.

    Very well said. I am experiencing this first hand. I built a product to scratch my own ‘goat’, and I am trying to learn about usage passively. My message has not been clear enough for my early testers to respond accordingly, and I haven’t been prepared to focus on one specific hypothesis.

  3. John McFarlane says:
    11.03.2012.

    Im glad that you pointed out # of sign ups? – A vanity metric.

    Im suprised that so many use this approach and either have a very vague description on a landing page with a mailing list subscription form or no info at all.

    Its almost funny how many have the typical “we’ll let you know when were ready” or something like that, but they haven’t considered whether people are ready for what they are building.

    Often the time between their email collection and anything being ready to test is so long that when i get an email letting me know that i can create an account that i might still remember signing up for it but cant for the life of me remember what first interested me.

    Often the anti-climax too of not having a standard login option and forcing the facebook login on you.

    1. Tristan says:
      13.03.2012.

      Totally concur.

      It’s very hard to go from a smoke test to real code and we all tend to get locked into a massive build after MVP. There needs to be an MVP 1.1

  4. Enny O says:
    10.05.2012.

    Hi Tristan,

    I could boast to be an expert and know all but am not. Thanks for this post. However, I’d like to ask between learning about one’s product and testing in the implementation of the Lean Startup, at what point should I be running to register my company name or betterstill, the patent and copyright?

    If I had to be practically lean in my works, I wouldn’t want to be running to the register a new business name everytime the former wasn’t viable.

    What are your thoughts please?

    1. Ray says:
      23.06.2012.

      Enny, don’t do it until you have to (validations); IMHO not until you people start paying you for your produce, ie when you need a bank account

    2. Tristan Kromer (@TriKro) says:
      23.06.2012.

      I agree with Ray but would temper that by asking, “how much risk is there?”

      If you’re got unlimited funds, it is not risky at all to spend money on trademarks, domain names, lawyers for patents, etc.

      In the case of some deep technological invention, a patent might be valuable down the road (or as social capital to be able to say, “Holder of 5 patents”) even if the business you were trying to execute on top of it was worthless.

      Personally, I am quick to buy domain names and slow to register trademarks. Copyrights are cheap and I used to do this immediately on writing a new song, but for a website, just write (c) copyright on the bottom. I never bother to register a company name, but I keep one LLC and do all my new business experiments and consulting under that LLC for liability protection.

  5. Hashim Warren says:
    12.09.2012.

    Where is Build-Measure-Learn reversed in Eric Ries’ book?

    1. Tristan says:
      12.09.2012.

      p. 78 and again on 201-202. Maybe some other places but I only just looked quickly in the index.

  6. Pingback: Survey Results: Are Developers More Productive In Scala? | Evolvable Me

  7. Salman Merchant says:
    14.03.2013.

    Doesn’t some small portion of your testing need to be with other customer segments, so you can see if some other part of a population besides your targeted customer segment is interested in your value proposition? Especially if you’re ONLY targeting your hypothesized target customer?

    1. Tristan Kromer (@TriKro) says:
      14.03.2013.

      I prefer to test segments in serial rather than parallel. If you test at the early stages in parallel you risk not having sufficient sample size to make an accurate judgement on any segment.

      In addition, it tends to fracture team focus.

      Better to stack rank the segments you’d like to focus on and then systematically test one after another.

      If the first one works, that doesn’t mean you can’t test an alternative segment to see if it’s even better.

  8. Evan L says:
    09.05.2013.

    I think this is a great wakeup call for entrepreneurs who misinterpret the lean startup methodology. However, I don’t agree that the cycle should be reversed.

    I have learned that the problem lies with the scale on which we perform the cycle. In fact, you outline the first full cycle in step 1 (learn): “build an experiment to test the hypothesis.” People tend to run away with the “build” part of the cycle instead of iterating at this scale (they fire cannon balls instead of bullets).

    From another angle: If your startup is simple enough to jump into learning before an experiment is performed, think bigger! This world has enough mobile payment systems, crowd funding platforms, and social media sites for pets as it is.

    1. Tristan says:
      09.05.2013.

      Speed is a great asset, but great speed in the wrong direction doesn’t get you to your goal.

      Putting learn first doesn’t mean “learn before doing”. It means figure out what you want to learn.

      In other words, steer in the right direction before you put your foot on the gas.

    2. Dan says:
      19.08.2013.

      I agree, I think there’s room for both. Whenever implementing a test in your product, it’s important to be clear about what results you expect and to make sure you can accurately collect data to compute your metrics. That said, there are times when you implement a test and you don’t get the expected results, but if you’re measuring everything you can notice unintended effects – very much like serendipity in scientific research (penicillin, teflon, viagra, etc).

      Tristan, if you’re doing rapid iteration right, if you’re moving in the wrong direction you can correct your path very early, and you won’t be moving in the wrong direction very long.

  9. Pingback: Procto » Articles » Minimum-Viable-Product Demystified (Part 2)

  10. Pingback: It’s MAD to Have a Separate Discovery Phase | Agile Product Discovery

  11. Pingback: It's MAD to Have a Separate Discovery Phase

  12. Pingback: The Ultimate Guide to Running a Lean Startup by @TriKro

  13. Pingback: The Rudder Fallacy – Adopting Lean Startup Principles | Digital Marketing Blog

  14. Pingback: It's MAD to Have a Separate Discovery Phase - Aaron Sanders | Agile Coach

  15. Pingback: The Rudder Fallacy – Adopting Lean Startup Principles | Cornel Lazar

  16. Pingback: Creating a culture of learning at The Guardian: our Product and UX Toolkit – Charlotte Gauthier

  17. Pingback: Lean validation: past, present and future

  18. Pingback: Iteration = Time to Learn, Not Time to Build

  19. Pingback: S.I.M.P.L.E. before S.M.A.R.T. – Thoughts from a Lean Startup Coach & Innovation Ecosystem Designer

  20. Pingback: Fear and the Sandbox, Part 1 – Thoughts from an Innovation Coach & Ecosystem Designer

  21. Pingback: Our New Lean Experiment Template (and Why You Shouldn't Use It) – Thoughts from an Innovation Coach & Ecosystem Designer

  22. Pingback: Adoptando el Lean Hype Cycle (Ciclo de Sobreexpectación)

Got something to say?

This site uses Akismet to reduce spam. Learn how your comment data is processed.