The staff survey is almost ubiquitous in the modern workplace providing a broad range of information for consultants, HR practitioners and managers. Almost universally the collection of employee responses is undertaken through Likert-style, attitude questions. Whilst such data contributes to understanding the workplace, a largely untapped data source which would provide greater richness of analysis is available through the inclusion of free-text questions in staff surveys. These, however, pose methodological and logistical difficulties in a very large staff survey due to the sheer volume of data available. This paper compares two ways of analysing such data, suggesting that automating analysis can make the use of this data advantageous for both managerial and research outcomes.
First we describe the development and application of targeted, free-text response questions for use in a staff survey administered in a large public sector workforce (n = 102,219). We then focus on comparing results from manual and automated analyses of one question in the data that generated over 180,000 distinct responses.
The manual approach involved coding the data inductively, identifying what emerged from the data itself. This was then compared with the findings from an automated approach which used the qualitative data analysis software Leximancer to identify common concepts in the data which are then grouped into consistent themes. It became apparent that the application of Leximancer to survey data was atypical and we present some methodological challenges which required a sophisticated understanding of both the tool and the research problem.
The comparability of the results across the two methodologies will be discussed, as well as the benefits and disadvantages of each approach. Comparison of the two analyses showed broadly comparable results at the thematic level with some nuanced differences at the micro level. We conclude that the utilisation of either approach for very large datasets primarily depends on the purpose of the analysis as, although the manual approach can give the researcher a more detailed understanding of the data, for the very large data volumes typical of a staff survey, it presents substantial logistical challenges. The comparability of the results at the thematic level suggests that, for very large datasets, an automated approach is more practical and can provide useful indicators of content which can then be followed up in more depth if required.