S.I.M.P.L.E. before S.M.A.R.T.

If you’re a regular reader, you may know that we have a somewhat complicated template for designing effective experiments. You may also know that we recommend everyone ignore it and make their own. We’re funny like that.

We love our template and we use it regularly, but for a team new to the principles of Lean, it is complete overkill. Designing an effective experiment is a daunting and time-consuming process, so we often design “bad” experiments without using a template just to get the teams started.

Velocity before Insights

One of the principles we focus on when coaching a new team is based on the Rudder Fallacy. A rudder is useless for changing direction if the boat isn’t moving. So the first thing we have to do to navigate is start rowing.

lean startup crew boat
With an innovation team, it’s better to start running experiments than to worry about whether we’re running the right experiment or a perfectly designed one. Click To Tweet

We encourage teams to get out of the building, listen to customers, and run quick landing page tests — anything to get them moving forward.

We’d rather teams run lousy interviews, ask all the wrong questions, and come back confused than spend a week designing a “perfect” customer persona and interview guide without actually speaking to any customers. Often, even with terrible interview techniques, a few smart innovators will return with the revelation that they should question their assumptions. And that’s the only insight they need: that they can be wrong, that they don’t know as much as they think they do.

So just as an MVP with customer feedback beats a “perfect” product that has never touched a user, a half-baked experiment beats a perfect one that has never been run.

Starting Simple

So we often don’t use our complicated experiment template when working with a new team. We just talk through the Build, Measure, Learn loop.

Build Measure Learn Loop

But we do it backwards. Because we’re funny like that too.

Learn – What are we trying to learn? What is the one question whose answer would offer progress to our business model?

Examples:

  • Who is our customer?
  • How much will they pay?
  • What features would solve their problems?

Measure – What data, qualitative or quantitative, would help answer that question? Do we need a yes or no answer to a question? Do we need to see someone make a purchase? Do we need to see a customer’s expression of delight when using our product?

Examples:

  • Percent conversion rate on a landing page
  • Qualitative observations and video of a usability testing session
  • Market research data on the total addressable market

Build – What do we need to do to get that data? What is the least amount of effort we can put in to get an answer to our question?

Examples:

This simple formulation can be quickly grasped and used to design a basic experiment without worrying about the early stop condition, the fail condition, or anything else that would distract the team from getting out of the building.

A coach with a cap and a whistle

(Of course, a coach should still address issues such as setting a clear time box by asking the team about their experiment’s time frame.)

Learn Simple Template

Which brings us to a simpler template that we designed with only four boxes. Again, we go backwards through Build, Measure, Learn, adding a stopping point called Results & Insights.

  • Learn -> Learning Goal – What one, Specific question about our project should we answer that is Relevant to making progress?
  • Measure -> Data – What Measurable qualitative data or quantitative metrics will we collect?
  • Build -> Plan – How and when will we collect the data in a Timely fashion? Is it Achievable?
Graduation cap

With the addition of Results & Insights, we’ve concluded our experiment. The Build, Measure, Learn loop may go round and round, but the experiment will stop once we have data that we can act on.

Simple = S.I.M.P.L.E.

Since our other template is called Learn S.M.A.R.T. (Specific, Measurable, Achievable, Relevant, & Timely), we decided to call this template Learn S.I.M.P.L.E. (Small Insights Mean Progress with Lean Experiments).

Kromatic - Learn S.I.M.P.L.E. Template
https://www.kromatic.com/resources/learn-simple

Now have fun and get out of the building!

Lessons Learned

  • Out of the building is better than a perfect experiment never launched
  • Don’t be a slave to templates
  • Start S.I.M.P.L.E. and then figure out how to be S.M.A.R.T.


Discussion (3 comments)

  1. Pingback: S.I.M.P.L.E. nên trước S.M.A.R.T. – TBI | Technology Business Incubator Center

  2. Andy Cars says:
    13.02.2019.

    Thanks Tristan for publishing value adding and original content on a consistent basis. I always enjoy reading your posts.

  3. Wendi Bukowitz says:
    20.02.2019.

    Timely reminder that getting outside the echo chamber of in-house experts is the way to jumpstart an innovation mindset

  4. Pingback: Friday Five: March 1, 2019 - Annie S. Anderson

  5. Vincent Ducret says:
    05.03.2019.

    Thanks Tristan for this article. Don’t you think that omitting an Invalidation Criteria field in SIMPLE template might be an issue for quantitative-date related experiment? How might team take decision in such case if they don’t define in advance success or failure criteria for their experiment?

  6. Pingback: S.I.M.P.L.E. nên trước S.M.A.R.T. | FINNO

Got something to say?

This site uses Akismet to reduce spam. Learn how your comment data is processed.