Writing Strong and Effective Surveys

Every month we dive into what we are all so passionate about here at FMP: human capital management. With so many people working from home during the pandemic, employers and brands across the globe are sending more and more surveys to help take a pulse on their employees’ and clients’ needs, attitudes, and feelings. Unfortunately, not all surveys are created equal and we’ve seen some lately that could have definitely used a helping hand in developing strong, reliable questions that will lead to concrete data. Here at FMP, survey development is one of our strongest capabilities, and we’re happy to share some tips for how to develop solid survey questions for use in your organization.

Figure out what kinds of questions you want to ask: The most significant decision you can make in developing a survey is whether you want questions posed as open-ended (where respondents answer in their own words) or closed-ended (where they are asked to choose from a list of answer choices). Closed-ended questions require a respondent to choose an answer from a pre-set list, which can make it easier for analysis. However, depending on the question, some respondents may find that their ideal choice is not on the list. Open-ended questions allow the respondent to answer any way they like which gives you more detail – however, it often takes more time and effort to analyze qualitative data, and without discrete answer guidance, your list of responses may grow out of control. A happy medium is to use closed-ended questions with an “Other” option at the end of the answer list.

Keep it short and sweet: In an age where data is king, you may be tempted to ask as many questions as possible to get as much information as you can from your respondents. However, survey fatigue is a real issue, and if your survey is too long or too complicated, you run the risk of having respondents drop out partway through. Keep the questions simple and short and be mindful of how many questions you ask and how long it will take your respondents. Most online survey platforms will give you an estimate of how long your survey will take to complete, so heed the advice. It may also be beneficial to add a progress bar to your survey, so respondents know approximately how much longer they have left to go to the end.

Pay close attention to your wording: Ask questions that are clear and specific with simple words. Make sure that you are only asking one thing at a time and avoid “double-barrel” questions; for example, if your question reads “Are these meetings interesting and helpful?,” respondents who think the meetings are interesting but NOT helpful will have a hard time answering and your data will suffer. In this example, it would be more effective to ask two separate questions.

Develop your answer choices wisely: Most of the time, the number of answer choices should be kept to a minimum – four to seven options at most. Research indicates that people have a hard time keeping more than this number of choices in mind at one time. An exception to this is when the question is asking about an objective fact, such as the birth month or religious affiliation of the respondent. Generally, providing a middle category (an odd number of choices) provides better data. Consultants often use a Likert scale, a five- or seven-point scale that allows respondents to choose how much they agree or disagree with a particular statement. Points on the scale should be labeled with clear, unambiguous words such as “Agree,” “Somewhat Agree,” or “Neutral”.

Be mindful of question order: Things mentioned early in a survey can impact answers later; if the survey mentions something specific and easily remembered, respondents might be primed to think of this when answering other questions. If you can, randomize the order that questions appear to your respondents. Ask sensitive questions like demographics at the end of the survey.

Pre-Test your survey: Before deploying, send your survey to a few potential respondents or your colleagues to get feedback. Ask your test group to read the questions to make sure nothing is confusing or unclear. Which leads us to our final tip:

Evaluate each question: The ideal question accomplishes three goals: 1) It measures the underlying concept it is intended to, 2) It doesn’t measure other concepts, and 3) It means the same thing to all respondents. Go through your questions one by one and ensure all of these criteria are met.

Survey design is both an art and a science, and the more you do it the better you get at it. Follow these key tips for developing your survey questions, and you’ll be well on your way to making clear, data-driven decisions in your organization.

Interested in learning more about how to design effective surveys? Reach out to us at bd@fmpconsulting.com!

Alexandra Flagg joined FMP in March 2019 as an I/O Psychology consultant and spends much of her time geeking out over survey design and methodology. Alex fuels her love of surveys and data with her addiction to iced coffee.