Survey design is a critical part of survey research. Seemingly simple characteristics can have pronounced effects on participants and how they respond to items. However, it is often difficult to anticipate what effect these characteristics can have. One such characteristic is the response options participants use to respond to items.
The Australian Public Service (APS) carries out an annual, large scale employee survey: the APS Employee Census. This has provided the opportunity to test how participant responses are influenced by different sets of response options. Historically, the Employee Census has used a five-point Likert scale including a neutral mid-point and, in some cases, a ‘Not Sure’ option. However, 5% of respondents were randomly selected and streamed into an alternate version of the census providing the same items with a different response scale. In this version, the neutral mid-point and the Not Sure options were removed. This allowed comparisons to be made between respondents from the same sampling frame answering the same questions, with only the response options varying.
The two response scales were compared in four ways to determine whether changing the response options led to a) changes in the proportion of participants skipping items; b) changes in agreement rates; c) changes in skewness and kurtosis of item responses; and d) changes in inter-item correlations.
Results suggest on average the number of participants skipping each item increased by less than 1.5%. Furthermore, when the mid-point was removed, respondents were more likely to respond positively. In some cases this effect was quite large, with respondents more than 50% more likely to agree. Changing the response scale also led to reductions in skewness, although changes in kurtosis were relatively small. Changes in inter-item correlations were also relatively rare, although is some cases the inter-item correlations between items with the 6-point scale were stronger.
This demonstrates that a scales response options can have significant impacts on some survey responses, although others may be less impacted. However, it is critical to take into account whether neutral mid-points and ‘Not Sure’ options are appropriate on a case-by-case basis because of the difficulty in determining which response options provide a more ‘accurate’ measure.