Let’s Play: What’s Wrong With This Survey

by | Research

An early-stage investor in Chickety Split, a healthy ready-to-eat food product that you can buy in grocery stores, recently asked me to take a look at a survey that the founder, Fred, is preparing to send out.

The investor, Harry, explained that he was seeking my advice because he expected that the survey would result in a disappointing outcome for Fred. However, with no experience in survey construction, Harry was having a hard time providing specific actionable feedback to improve the survey and its results. My first question for Harry was: What is Fred’s survey objective? Whenever you do research, this is a key first step. It’s important that you have a question that requires specific, statistically significant data. Fred’s objective was to get insight into how Chickety Split is used by consumers. 

With this “soft” objective, qualitative research would be a better choice, as that’s what’s generally used for gaining deeper consumer insights. Plus, I’m guessing that Fred only has a few hundred people in the Chickety Split database. That’s not enough to get statistically significant results. Fred may want to consider interviewing actual customers instead of sending out a survey. By doing this, he’ll get a richer, more in-depth understanding of his target consumers, which he seems to need.

After reviewing the survey, I agreed with Harry’s overall concerns. I don’t think the results are going to help Fred make any business decisions. 

My email to Harry:

You’re right, Harry, there are a lot of problems with the survey as it’s written. The problems include 1) ambiguous wording, 2) two questions in one, and 3) answer choices that are uneven and sometimes don’t even align with the question.

But let’s start with a couple of kudos:
  1. None of the questions of the Chickety Split survey are mandatory. This is a great move! By making a question mandatory, you’re basically saying that the question is so important that if they don’t answer that question, you don’t want to hear their answers to any of your other questions.
  2. Each of the multiple choice questions includes an option for “Other, please specify.” This is smart, because there’s no way Fred can know all of the possible answers. The key is to list the most likely ones for easy analytics but still allow room for someone to manually enter a different option.
Five key suggested changes:
 
Question 1: When I see a post on social media from a food brand I buy, I care most about seeing (check all that apply):
– Recipe ideas
– Story behind founder or product
– Where to find the product
– Testimonials on the product
– Tips about how to eat/live healthier
– Coupons or giveaways
– Other (Please specify)
 
Feedback:
– Have the respondents rank the choices in order of importance to them instead of just checking all that apply
 
Question 2: How do you decide between your meals on the go?
– Only choose healthy options
– Sometimes choose healthy options
– Choose the most convenient option
– Cheapest
– Choose the tastiest option
– Other (Please specify)
 
Feedback:
– Before Fred asks this question, he should ask a separate question about frequency: How often are you eating meals on the go?
– What does “decide between” mean here? Would everyone who reads this question understand it in the same way? Avoid ambiguous words and phrases.
 
Question 3: What makes a meal healthy? (Check all that apply)
– 800 calories or less
– Organic
– No preservatives
– 28 g of protein
– Locally sourced
– No hormones
– Fat free
– Other (Please Specify)
 
Feedback:
– I’m not sure that Fred will learn anything here that’s actually actionable.
– Plus he’s really put himself in a corner by using specific, fixed numbers. For example, what if they think a meal with 750 calories is healthy but a meal with 800 calories is not?
 
Question 4: Have you bought Chickety Split?
– Haven’t tried it
– Bought it once but not again
– 1-2 times per month
– 3 or more times per month
 
Feedback:
– This should be a Yes/No question
– The answer choices are not consistent – two are about frequency of purchase and two are about if they’ve purchased
 
Question 5: Would you refer Chickety Split to a friend?
– Yes
– No
– Other or Not Applicable (Please Specify)
 
Feedback:
– How do you refer a meal to a friend? Use clear, specific language.
– How could there be an answer other than yes or no?
– Asking this kind of referral question is irrelevant unless it’s actually a Net Promoter Score question. In that case, the question must use the exact wording and answer scale.

Fred’s mistakes here are ones we see new entrepreneurs making over and over when it comes to survey research. But you can see how precision can matter a lot when it comes to asking questions in a way that will provide real, actionable data.

Just because survey tools make fielding quantitative research easy, doesn’t mean that it’s easy to do this kind of research. That’s why I often recommend that entrepreneurs without real research experience stick with qualitative research early on as they’re learning about their customers and the marketplace. If you want to learn how to ask great qualitative questions, pick up a copy of Talking to Humans by Constable and Rimalovski. It’s a “practical guide to the qualitative side of customer development.”