Comparing the quality of data from postal and online questionnaires
Building: Holme Building
Room: MacCallum Room
Date: 2010-12-02 03:30 PM – 05:00 PM
Last modified: 2010-11-17
Abstract
Introduction: Paper-and-pencil questionnaires returned by post (postal questionnaires) have traditionally served as a primary data collection method in social science research. However, the use of computer-based questionnaires completed on the Internet (online questionnaires) is becoming commonplace. Despite their widespread use, the quality of online questionnaire data has not been comprehensively investigated. Data quality may be affected by both measurement and non-measurement error. Numerous within-subjects studies in which participants completed paper-and-pencil and computer-based versions of the same questionnaire have demonstrated the equivalency of data collected using both questionnaire modes. Therefore, measurement error does not appear to differentially affect the quality of data collected via computer. However, these studies do not take into account that use of different questionnaire modes may introduce non-measurement errors such as participant non-response, item non-response, and selection bias. Although several studies have compared the quality of data from telephone, face-to-face, and postal questionnaires, few studies have compared postal with online questionnaires.
Aims: To compare the quality of data from postal and online questionnaires.
Methods: This presentation compares the responses of men taking part in a study investigating the psychosocial sequelae of testicular cancer. Participants could choose to complete the study questionnaire on paper or online. Outcomes of interest for the current analysis included participant non-response, item non-response, and selection bias. Additionally, the practicality of the different questionnaire modes was assessed by calculating the time taken and number of reminders required for questionnaire return.
Results: There was no significant difference in participant non-response according to questionnaire mode; 77/100 (77%) of postal questionnaires were returned, versus 62/73 (85%) of online questionnaires (p=.24). There was also no significant difference in item non-response according to questionnaire mode; 34% of returned postal questionnaires had missing items compared with 36% of returned online questionnaires (p=.97). Additionally, there was no significant difference in the average number of items missing from the 215 item questionnaire according to whether the postal (M=2.9 items per participant, SD=13.6) or online (M=1.7, SD=4.7; p=.274) version was completed. A significantly greater proportion of participants who completed the online questionnaire had tertiary education (44% vs 26%; p=.046) and worked in highly skilled occupations (62% vs 39%; p=.016), whilst a significantly lower proportion had a partner (58% vs 75%; p=.048). There were no significant differences in participants who completed postal and online questionnaires according to age (p=.90), country of birth (p=.92), primary language spoken (p=.86) or employment status (p=.27). Online questionnaires were returned significantly more quickly (M=19.6 days after questionnaire delivery, SD=36.6 vs M=38.1, SD=35.9; p=.003) and required significantly fewer reminders (M=1.0 phone or email reminder, SD=1.4 vs M=2.5, SD=2.5, p=.001) than postal questionnaires.
Conclusions: Online questionnaires offer greater expediency than postal questionnaires without any detrimental impact on data quality due to participant or item non-response. However, investigators should be aware of potential selection bias created by more single and highly educated people completing online questionnaires compared with postal questionnaires. Besides this issue, online questionnaires offer an efficient method for collecting high-quality data, with faster return and fewer reminders.