Evaluative Research Of Customer Development Mistakes

I realized today that I have been taking the lean startup / customer development axiom of “talk to customers” far too literally.While the quality of my feedback (evaluative research) has been reasonably good and I’ve gotten a number of good ideas, there’s a big difference between talking to customers and listening to customers.

I’ve been having far to much of a two way dialog and not letting my product speak for itself.

Screen-shot Demo

I got a call today from Performable.com which is developing a new A/B testing tool for this type of evaluative research. I signed up for their beta test a few weeks ago and they gave me a call to ask if I would be willing to walk through a demo with them.

Within a minute, they had a screen share up and were showing me a live demo. Although I had no control, it was definitely not a walk through in the traditional sense. it wasn’t a guided tour so much as an exploration. They showed me a screen and started peppering me with questions. Some of my favorites:

  • What do you think this page does?
  • What do you think will happen if you click this button?
  • What do you think this wording means?

Within a few minutes they had me dictating every thought running to my head and quantifying exactly what I needed from an A/B test tool. At no point did they resort to telling me what the page was supposed to do. In truth, I don’t really even know if the product worked or if I was just looking at static pages.

That’s exactly the sort of feedback I need.

Big Ears

Getting back to work I decided to immediately get some customers and ask them about my Photoshop mock-up of a user’s profile page. This is the mock-up that I thought was “good enough” and that both my partners, a professional designer, and two customers had already looked over. At least for version 0.1, I thought my customer development was done.

I was wrong.

One of the very first things that the customer asked was, “How do I contact this person?”

After spending more time on this than I’d care to admit and four sets of eyes staring at this page multiple times, we had forgotten the most basic functionality of all! There was no space for a contact / connect button. Doh!

Of course, customer development is an ongoing process and there is no “done”, but this was an embarrassing omission and spoke to something more fundamentally wrong in my customer interview technique. The big mistake was that I had previously performed more of a walk through, explaining each section to users and guiding them through.

This time I said less, and listened more. I didn’t offer any leading questions until they stopped talking, and then I tried to say very little aside from, “What about this?” “What about this section?”

I won’t go into the rest of the embarrassing elements I missed, but the change in customer interview style was very rewarding. So kudos and thanks to the Performable.com team!

Discussion (4 comments)

  1. Michael Fitzpatrick says:

    Small Ears, 😉

    I’ve used a couple of other tactics in the past to really ensure I’m letting the customer talk:

    – If using a presentation deck, start with a quick intro and brief (very brief) context of what general problem you are trying to solve. Then flip to the next slide, which should have a title along the lines of “Problems In this Area:”, and make sure it starts blank. Leave it blank for as long as possible and ask the customer/user to describe the problems they see in this area. You might then choose to show some of the problems you have identified, but only after the person you are talking to has exhausted their opinions.

    – Re: Photoshop. I love it, have used it for years, and it is so tempting to use for mockups and design iterations, but: most of the time, its inappropriate for this stage of development. The majority of photoshop mockups I’ve seen have pretty colors, gradients, and other treatments that completely distract (I prefer green vs. blue, can this be a rounded button…) from what you really want to know: does this mockup address the features the user identifies as important, and solve real problems.

    Good luck on this project, exciting!

  2. Tristan Kromer says:


    Surprisingly I haven’t gotten any comments on colors (except for my co-founders who don’t like orange). I’m kept it pretty monochrome to avoid such discussions and managed to get some good feedback on layout while avoiding minutia.

    I used to love ppts, but I’ve found that when I put one on people expect me to go into a long spiel of some sort which is what I’m trying to avoid. Have you had any good experience in avoiding this?


  3. Sean Murphy says:

    I would be asking more questions up front before you show them the mockup to make sure that you understand their situation. Probe for pain points that you believe Startup Square may address for them so that you can ask “will this help you with X” instead of “do you think you will use this.”

    A lot of times a drawing of their problem that they can markup (or in a shared drawing tool like http://dabbleboard.com/ where you can start with a drawing and “pass the pen” on-line) is more useful to better understand the problem then demoing your mockup.

    To me you are still at the “what problem are you solving, and how do I know that I have it” stage rather than “please give detailed feedback on how useful is this site is.”

  4. Tristan Kromer says:

    That’ll probably be my next “how I screwed up” blog post. 😉

    Although in my defense I should perhaps of mentioned that I am discussing mockups with people who have already identified that they have a problem finding and screening potential co-founders.


  5. Pingback: How to Practice Shutting Up (Customer Development Practice) | startupSQUARE's blog

  6. Pingback: Iterate Before you Iterate | startupSQUARE's blog

  7. Pingback: Clever Hans and Photoshop Iteration | startupSQUARE's blog

  8. Pingback: How to Practice Shutting Up (Customer Development Practice) | GrasshopperHerder.com

  9. Pingback: Clever Hans and Photoshop Iteration | GrasshopperHerder.com

  10. Pingback: Pay No Attention to the Man Behind the Curtain | GrasshopperHerder.com

  11. Pingback: Top Takeaway - Startup Lessons Learned | GrasshopperHerder.com

  12. Pingback: Hope is the Enemy - Startup Advice from a Monk | GrasshopperHerder.com

Got something to say?

This site uses Akismet to reduce spam. Learn how your comment data is processed.