If you’re a regular reader, you may know that we have a somewhat complicated template for designing effective experiments. You may also know that we recommend everyone ignore it and make their own. We’re funny like that.
We love our template and we use it regularly, but for a team new to the principles of Lean, it is complete overkill. Designing an effective experiment is a daunting and time-consuming process, so we often design “bad” experiments without using a template just to get the teams started.
Velocity before Insights
One of the principles we focus on when coaching a new team is based on the Rudder Fallacy. A rudder is useless for changing direction if the boat isn’t moving. So the first thing we have to do to navigate is start rowing.With an innovation team, it’s better to start running experiments than to worry about whether we’re running the right experiment or a perfectly designed one. Click To Tweet
We encourage teams to get out of the building, listen to customers, and run quick landing page tests — anything to get them moving forward.
We’d rather teams run lousy interviews, ask all the wrong questions, and come back confused than spend a week designing a “perfect” customer persona and interview guide without actually speaking to any customers. Often, even with terrible interview techniques, a few smart innovators will return with the revelation that they should question their assumptions. And that’s the only insight they need: that they can be wrong, that they don’t know as much as they think they do.
So just as an MVP with customer feedback beats a “perfect” product that has never touched a user, a half-baked experiment beats a perfect one that has never been run.
But we do it backwards. Because we’re funny like that too.
Learn – What are we trying to learn? What is the one question whose answer would offer progress to our business model?
- Who is our customer?
- How much will they pay?
- What features would solve their problems?
Measure – What data, qualitative or quantitative, would help answer that question? Do we need a yes or no answer to a question? Do we need to see someone make a purchase? Do we need to see a customer’s expression of delight when using our product?
- Percent conversion rate on a landing page
- Qualitative observations and video of a usability testing session
- Market research data on the total addressable market
Build – What do we need to do to get that data? What is the least amount of effort we can put in to get an answer to our question?
- Build a landing page
- Interview customers
- Offer the solution as a service instead of a product
This simple formulation can be quickly grasped and used to design a basic experiment without worrying about the early stop condition, the fail condition, or anything else that would distract the team from getting out of the building.
(Of course, a coach should still address issues such as setting a clear time box by asking the team about their experiment’s time frame.)
Learn Simple Template
Which brings us to a simpler template that we designed with only four boxes. Again, we go backwards through Build, Measure, Learn, adding a stopping point called Results & Insights.
- Learn -> Learning Goal – What one, Specific question about our project should we answer that is Relevant to making progress?
- Measure -> Data – What Measurable qualitative data or quantitative metrics will we collect?
- Build -> Plan – How and when will we collect the data in a Timely fashion? Is it Achievable?
With the addition of Results & Insights, we’ve concluded our experiment. The Build, Measure, Learn loop may go round and round, but the experiment will stop once we have data that we can act on.
Simple = S.I.M.P.L.E.
Since our other template is called Learn S.M.A.R.T. (Specific, Measurable, Achievable, Relevant, & Timely), we decided to call this template Learn S.I.M.P.L.E. (Small Insights Mean Progress with Lean Experiments).
Now have fun and get out of the building!
- Out of the building is better than a perfect experiment never launched
- Don’t be a slave to templates
- Start S.I.M.P.L.E. and then figure out how to be S.M.A.R.T.