Surveys can help your company identify opportunities, assess challenges, and set direction. If you're looking to design a survey for your own organization, you should be aware of common pitfalls that novice survey creators can run into.
Here are three survey question development mistakes made by startups and well-established companies alike. Caveat lector, or “let the reader beware”.
The Biased or Leading Question
“Our product reduces your tension by 10%. Would you like to buy it?”
This rhetoric might be fine for a home gym infomercial, but the question above is biased. A biased survey question prompts or leads a respondent to respond in favor or against a specific outcome, resulting in inaccurate data. The example above mentions a reduction in tension that might influence the respondent to indicate they would make a purchase. A better question would be "How likely are you to buy this product?" Examples don’t need to be as extreme as the one highlighted to be considered biased.
Tips to avoid bias:
- Use neutral language. Do not favor one option –explicitly or implicitly –and do not lead a respondent to an answer.
- Vary the ordering of options in a list. This should be done across questions with similar answers to ensure a respondent will not make a decision based on chronological sequencing.
The Ambiguous Question
“How do you feel about your purchase?”
This question uses ambiguous and imprecise language. Quantifying or assessing subjective attitudes is difficult, and the burden shouldn’t fall on the respondent. A better approach is to provide options for the respondent: "How satisfied are you with your purchase? Extremely satisfied; somewhat satisfied; neutral; somewhat unsatisfied; extremely unsatisfied".
Tips to avoid ambiguity:
- Think critically and develop precise questions. Even better – don’t reinvent the wheel and look at existing metrics, such as those available at Survey Legend.
- Consider tools other than surveys. Conjoint analysis, for example, can help reveal how much a customer values a certain product or service.
The Complex Question
“If you had to get to work using a bicycle, bus, train, car, or on foot, which would you choose? Consider annual precipitation, your transportation budget, and carpooling opportunities.”
The question above assesses propensity for transportation methods, perhaps to help a city decide whether to allocate funding to bike share programs, commuter rail services, or bus routes. However, this question is overly complex, making it difficult for the respondent to answer.
Tips to avoid complexity:
- Trim the fat. Cutting unnecessary qualifiers could ease the intellectual burden. Understand, however, that you risk perpetuating the second question type by making assessments more open-ended.
- Qualitative interviews and focus groups are an alternative method of receiving detailed answers if time and effort allow. A pair-wise ranking system might also be appropriate in this scenario.
When your company conducts surveys, look for red-flags that could lead to inaccurate conclusions. Low response rates or an unrepresentative sample of respondents are especially sharp objects that could poke a hole in your metaphorical tire. These three tips are only the beginning of becoming a proficient survey designer. Knowing the best way to design assessments – and being aware of when other tools may be more appropriate – will lead to more effective company research.
Interested in learning more about survey design and other principles of Business Analytics? Take HBX CORe to strengthen your knowledge of Business Analytics, Financial Accounting, and Economics for Managers.
About the Author
Anna Vallee is a Research and Teaching Assistant for the Business Analytics course at HBX. She received her Ed.M from the Harvard Graduate School of Education in 2015 where she studied experimental and quasi-experimental research design, applied data analysis, and management practices related to non-profit and educational institutions. Prior to joining HBX, she was the Manager of Research and Data Analytics at another Boston-based edtech startup. A lifelong learner, she is always looking for a great book to read.