Surveys in the Virtual Workplace: Tips for Addressing Common Issues

With the rise in virtual work brought upon by the COVID-19 pandemic, it has become increasingly crucial for companies to effectively leverage online surveys.¹ Although surveys have many advantages, they also have some drawbacks which, if not addressed, can limit their usefulness. Knowledge of common survey issues and how to address them is crucial for companies to get the most out of their surveys and engage in effective data-driven practices. Below, we discuss some common problems associated with the use of surveys and offer some tips for addressing these issues. 

COMMON SURVEY ISSUES AND WHY THEY MATTER 

• Nonresponse Bias: Ideally, everyone who is sent a survey will complete it. However, this (sadly) never happens in practice. Nonresponse bias refers to when a significant portion of people who are sent a survey do not respond.² This is problematic because it means your survey results may not be generalizable to everyone and that some employee voices are not being heard (e.g., if only leaders, but not entry-level workers, complete an employee engagement survey). 

• Survey Attrition: While someone may begin a survey, for one reason or another, they may decide not to finish it (e.g., the survey is too long). Attrition refers to when people drop out of a survey after providing some initial data.³ Much like nonresponse bias, attrition is problematic in that it can negatively impact the generalizability of your survey results.  

• Careless Responding: In some situations, people put little thought or care into their responses, but still, nonetheless, complete a survey. This is referred to as careless responding (e.g., imagine someone selects “strongly agree” for every item without actually reading anything).⁴ These responses are sneaky because they look meaningful, but are not. Careless responding is an issue because it misrepresents survey results and can cause you to make incorrect conclusions about your data. 

• Survey Fatigue: Survey fatigue refers to when people become exhausted from completing surveys.⁵ This is especially common when companies overwhelm their employees with too many survey requests (despite maybe having good intentions in doing so). Survey fatigue is a problem because it can lead to all of the above issues!    

• Social Desirability Bias: When completing surveys, people may respond in a way that makes them look good, rather than respond truthfully. This is referred to as social desirability bias (also called “faking”).⁶ This is especially common for sensitive topics where there is an incentive to provide inflated responses (e.g., self-ratings of an employee’s job performance, which may be linked to pay/promotion decisions). Like careless responding, social desirability bias can distort survey results and yield a false picture of your data. 

• Poorly Written Surveys: Up to now, the issues discussed have mostly been related to people’s survey response behaviors. Another common issue, however, is poorly constructed surveys (e.g., overly lengthy surveys, confusing questions). Writing effective surveys is an important first step for maximizing the usefulness of surveys and avoiding many of the issues noted above.⁷ 

WHAT CAN BE DONE? TIPS FOR ADDRESSING SURVEY ISSUES 

The good news is there are best practices you can adopt throughout the entire survey lifecycle that can minimize these issues. These tips are described in more detail below.  

• Take care in writing your survey questions: Arguably the most important part of survey development is crafting questions to accurately measure what you want them to. You want to make sure your survey data is built upon a solid foundation of clear and unbiased questions. For a great primer on how to craft strong and effective survey items, check out our blog post here! 

• Keep it short and sweet: There is a correlation between survey length and careless responding and attrition⁸ – the longer the survey, the more likely you are to have respondents who stop partway through or don’t take time to fully read or respond to the questions. By keeping the survey on the shorter side, you can cut down on many sources of bias. 

• Be prudent in what you include: When it comes to surveys, stakeholders can have all sorts of different opinions about what should be included. In these situations, it can be easy to give in and include everything but the kitchen sink. It’s important to balance what stakeholders want to see in a survey with what is really necessary to include. Ask yourself – what will we do with this information? If the survey item will not result in actionable data, consider leaving it out.  

• Consider anonymous or confidential surveys: When soliciting feedback or asking questions that may stir an emotion, allowing a level of anonymity or confidentiality is the best way to get someone to open up. In these types of questions, anonymity is key in convincing respondents to respond honestly, cutting the risk of social desirability bias. Communicate anonymity or confidentiality with the respondents – and then follow through. If you don’t feel you can successfully keep an internal survey anonymous or confidential, consider hiring a third party to administer your survey.  

• Be intentional in your scheduling: Receiving too many surveys at once can cause respondents to be overwhelmed and suffer survey fatigue. On the other hand, if you’re doing say, an employee engagement survey, administering it once a year may not be often enough for your organization to keep in line with trends or analyze the data over time. It is a myth that there is a one-size-fits-all approach to survey cadence – each organization is unique, and the survey administration should fit the situation.⁹ By consciously scheduling your survey at the right time for the organization, you will take a step in setting your survey up for success.  

• Share your results: This does not just mean sharing the excel file of all the data analysis (though you could do that if you wanted to). Share how the data you gleaned from the survey has led to actionable, measurable changes or results. This tip is especially important for employee engagement and pulse surveys – if employees see the same questions show up on surveys time and time again, with no indication that the organization is using the data to drive change, it erodes the trust between employers and employees. Think of a survey as a collaborative effort to improve. Participation is stronger when the organization has a good track record of sharing results and implementing changes.⁹  

Because of COVID-19 and the move to a virtual work environment, more and more organizations are seeing the benefits of implementing online surveys, and this will only increase as time goes on. These tips will help you minimize bias and other survey issues while maximizing your chances for actionable results and engaging in effective data-driven practices.  

What tips do you have for addressing survey issues? Share your thoughts with us on LinkedIn! 

REFERENCES  

1. COVID-19: Why you need to survey your employees right now (Article Link). 

2. Nonresponse bias: Definition, examples (Article Link).  

3. How to avoid survey attrition and keep sought-after respondents (Article Link).  

4. Identifying careless responses in survey data (Article Link).  

5. Survey fatigue: Everything you should know before creating your next online survey (Article Link).  

6. How to combat social desirability bias (Article Link).  

7. Writing strong and effective surveys (Article Link).  

8. The effects of questionnaire length and behavioral consequences on careless responding (Article Link).  

9. “Survey myths, decisions, & pitfalls” presentation at SIOP 2021. 

Nathaniel (Nate) Voss joined FMP Consulting as a Human Capital Consultant in May 2021. As an industrial/organizational psychologist, some of his favorite research areas include personnel selection, measurement, and survey methods. When he is not nerding out about one of these topics, Nate enjoys eating out at new restaurants and playing games with friends.      

Alexandra Flagg joined FMP in March 2019 as an I/O Psychology consultant and spends much of her time geeking out over survey design and methodology. Alex fuels her love of surveys and data with her addiction to iced coffee.