Top 10 Entrepreneurial Research Mistakes

The most common research mistakes I encounter fall into three main camps:

  • Research methodology selection
  • The questions asked and the answer choices provided
  • The care & feeding of respondents/participants

Research Methodology: Quantitative v. Qualitative

1. Thinking Quant is better than Qual

While achieving statistical significance can be very useful, most of my clients don’t have easy, inexpensive access to 600relevant potential respondents. That’s what would be necessary to get the bare minimum 30 responses required for results to be statistically significant, assuming a somewhat high 5% response rate.

In order to do even a modicum of segmentation with your responses, you really want to get at least 100 of them.  Doing the math, this means you’d need to start with 2000 names, again at a 5% response rate.

For most start-ups, and even many of my mid-sized clients, this makes quantitative research a non-starter unless they’re willing to pay for someone to source relevant respondents for them. And generally they are not! So while quantitative research can be very valuable because of the higher degree of certainty and ability to extrapolate, many times qualitative is the right way to go.

2. Research = Focus Groups

I wish I had a dollar for every time someone equated market research with focus groups. Everyone seems to think that research is simply a matter of “We’ll just get a few people together informally and talk to them.” Focus groups are just one methodology option and, for entrepreneurs, not necessarily a sensible one.

I often tell clients that there is a reason why research moderators make very good livings coordinating and moderating focus groups – it’s because it’s not easy to do it well!

On the flip side, I absolutely recommend having 1:1 conversations with clients and potential clients. The more that is learned directly from the horse’s mouth the better, and 1:1s are not only easier to coordinate, they’re easier for an amateur to conduct.

If you do conduct some 1:1 interviews, please remember that it’s not enough to have the conversation. You should write down what you hear and learn to try to ensure that the information remains unclouded by the effects of time and distant memory.

Questions and Answers

3. Asking “nice-to-know” questions

Don’t squander your time with someone who has actually opted to talk with you.  Make sure you only take up their valuable time with your must-know questions.  It’s a need-to-know question if you can clearly articulate what the answer will enable you and your company to do differently.

4. Mindlessly making your questions mandatory

Making a question mandatory is basically an indication that if the respondent doesn’t answer that one question, it’s not worth it to you to get their answers to any of your other questions.  In my experience, this is typically not the case. So avoid mandatory questions if you can.

5. Writing ambiguous questions and/or answers

This seems to be driven by two things:
A) Being too close to your subject matter and assuming that everyone sees the world the way you do;
B) Trying to limit the number of questions by mashing multiple questions together. Having the word “and” in a question is one indication that you may be trying to combine more than one question into a single sentence.

One way to check for ambiguity is to have someone you know take the survey.  Ask him or her to talk through their answers to each question as well as their understanding of what the question is asking. This should help you identify some of the ambiguities.

Here’s an example of an ambiguous question: How many technology professionals does your firm employ?  You need to realize that the way that you define “technology professional” might be very different from how your respondent defines it.

6. Assuming you know all the possible answers

Always add an “N/A” or “Other – Please specify” answer option to any multiple choice list. For example, one of my clients originally had this question on their survey: Please tell us your objective in participating in our program:

  • Weight loss
  • Get fit
  • Feel better
  • Visit Brazil
  • Personal Growth

The problem here is what if the respondent’s answer isn’t included in this list of choices?  Providing the answer option “Other – Please specify” not only gives voice to other perspectives, but may help you identify a new/different category for use in future research.

7. Playing fast & loose with Net Promoter Score

Ay yi yi.  I hate to even get started on this and am constantly appalled at what people do to this question.  Regardless of your opinion about Net Promoter Score (NPS), please remember that the scale must be 0-10 in order to be able to calculate NPS correctly.

While I disagree that the NPS is the only number you need to know (learn all about NPS in the original HBR article The One Number You Need to Grow), it can be a good indicator of brand health.  That said, it gives you no information regarding how to improve, and that’s why I recommend including it in a suite of satisfaction questions.

It is fun to have a brand health benchmark that you can compare to the scores of other companies.  For example, did you know that the median score (2013) was only a 16?!  At that time, Southwest Airlines had a 51, Apple had a 77 and Harley Davidson had the mind-blowingly high score of 81.

Respondents

8. Asking unqualified respondents to predict the future

Don’t ask broad and forward-looking questions of all respondents, such as: “What skillsets will be the most valuable to technology job seekers within the next five years?”

Instead maybe ask something grounded in what the respondent might actually have direct, credible knowledge. An example of this is: “What technology jobs is your company currently having a hard time filling?”  Another example is: “What new technology positions will your company be adding next year?”

9. Expecting respondents to create/iterate for you

In my experience, most people have a hard time responding to white space requests like, “What features should we add to our product?”  They are much better at reacting to concrete choices that you present, for example, “Please rank the following features in the order of your interest in them.”

While this does require that you and your team actually take the time to brainstorm and write up possible new feature options for parsing, that is your job.

10. Writing ambiguous questions and/or answers

As with any customer outreach, you need to have a clear picture of whom you are talking to.  Is it current users of your service?  Previous users?  At its most basic level, the wording of questions will vary depending on whether you’re talking in the present tense or past.  And it only gets more complicated from there.

Here’s a real world example of two questions on the same survey that are written for two different audiences: current users in the first and previous users in the second:

  • What is most intimidating or challenging about your current fundraising situation?
  • How much money did your fundraiser raise?

All in all, avoiding these common market research mistakes will not only help you make better use of your time and resources, it will get you information you can use to drive your company forward for future growth in a more informed way!