Comprehension vs. Commitment

Lemonade smoke test minimum viable product

I love smoke tests. They’re a great way of testing whether or not there is any serious demand for our value proposition.

One common form of smoke test out of the Unofficial Startup Real Book is a landing page test. The landing page states a value proposition with a call to action that asks the user to commit some form of value such as an email address or even money to sign up. Based on the % of users who signup, we get a rough signal from the market if the value proposition is in sufficient demand to build a minimal solution. If not, pivot!

But as much as I like smoke tests, I hate premature surrender. That’s what happens when the signal we get from the market is NO, but we don’t truly grasp why.

If you’re interested in cutting to the chase and reading the entire Real Book, you can download it here:

icon - download sumome

Landing Page Test

Let’s say we’re selling a new type of shoe that cures plantar fasciitis. We put up a landing page test with our value prop and a “buy now” button. Then we put $1000 in to Google Adwords for “shoes” to drive traffic and sit back to wait for the money to start rolling in…

Comprehension Test - cures plantar fasciitis

…and our conversion rate is 0%

Should we give up? That’s what the landing page test says. There is insufficient demand for this product.

By this point, some of you are probably asking, “What the hell is plantar fasciitis?”

Exactly. All those people coming to our site are asking the same thing.

Is our test failing because customers aren’t interested? Or do they fail because they simply can’t understand the value proposition?

Comprehension Test

Comprehension test - stop clubbing baby seals - Illustration by Emily Chiu

(Illustration and pun thanks to Emily Chiu)

 

Before we ever run a smoke test, we should run a comprehension test. It’s a simple test we can do that usually takes less than one hour:

  1. Write out our value proposition in 1-3 sentences.
  2. Show the value proposition to a participant for a few moments, just enough for them to read it. Then take it away.
  3. Ask them to explain the value proposition back to us in their own words.

If the participant’s explanation is roughly comparable to our own, we count that as a positive result. If not, then it’s a negative. For this sort of test, we generally want a sample size of about 20 people and a positive conversion of about 80%.

Here’s a simple tool to track testing activities.

The conversion has to be very high because regardless of what our value proposition is, people should understand it.

Who?

No customer persona, no one to comprehension testWho we bring to this sort of test is important, but we don’t actually need to use our target customer. We just need someone with the same basic education level and vocabulary. That’s because we are not testing to see if someone wants our value prop, just if they can understand it.

If our target market has a high school level education, then we can just walk out onto the street for participants.

If our target market has a specialized vocabulary such as Chief Marketing Officers, we can probably run a comprehension test with anyone with a marketing background. MBAs seniors might do just fine and they are a lot more accessible than CMOs.

Once we’re certain people understand our value proposition, only then can we run our smoke test and be confident about the results.

Qualitative vs. Quantitative

Comprehension Test: Qualitative vs. Quantitative?Don’t get caught up on qualitative vs. quantitative. Comprehension tests are both.

We get a yes/no quantitative response depending on the % of people that can successfully explain our value proposition.

We get the actual explanations as qualitative data. This data can be incredibly useful. If we use the comprehension test on our value prop “shoes that cure plantar fasciitis” we might get responses such as:

  • “Don’t know.”
  • “No idea.”
  • “Don’t know.”
  • “Something that cures a scary sounding disease.”
  • “Don’t know.”
  • “Shoes that help with a common runner’s condition that causes foot pain in the morning.”

Clearly the first five aren’t very useful, but the last one is. The last response is from someone who actually understood, probably a runner. Plantar fasciitis is a very common problem for runners.

Although our original value proposition failed the comprehension test, our respondent has given us a new value proposition to test!

Not only that, they’ve suggested some great keywords we can use for our Adwords campaign…something a lot more specific than “shoes” and with a lot more search volume than “shoes for plantar fasciitis.”

Rules of Thumb

  • Before testing commitment, test comprehension
  • Qualitative vs. quantitative is a useless distinction, we need both.

If you’re interested in reading the entire Real Book, you can download it here:

Learn more about comprehension tests

So…what should I post next? Tweet to tell me what to write:

Show me how to test product market fit!

or

How can I do lean startup in my friggin' huge company?

7 comments

  1. Todd says:

    Great article. Very informative and more importantly it provided me with actionable information as I move forward with the planning and development of my future businesses product lines.

  2. Caroline Bergaud says:

    This is what the survey industry has done for decades in ‘cognitive labs’ to test that their survey questions were received as they were intended, Tristan 😉 As a former Gallup director, this sounds very familiar and is equally important when you’re collecting data to inform policy-making. Just my 2 cents!

    • Tristan says:

      Excellent point! Do you have any articles we can reference to. I’ve been very surprised that this isn’t more widely done in the startup community.

      • Caroline says:

        Here is a doc from Pearson that introduces the concept, Tristan: http://images.pearsonassessments.com/images/tmrs/tmrs_rg/CognitiveLabs.pdf

        There are a lot of things that are not widely done in the startup community 😉 Despite claims to thinking out of the box, there’s a fair bit of conformity there. Just look at most startups’ websites… ;p

  3. Pingback: Websitetitel

  4. Pingback: 6 Reasons Why Your Conversion Rate Is Zero - The Asian Entrepreneur

Leave a Comment

Your email address will not be published. Required fields are marked *