In recent decades there has been an increase in the pre-testing methods used to assess the likely quality of survey questions. Whilst many scholars have written about designing and carrying out studies which involve these techniques (e.g. Campanelli, 2008), little is known about how effective each method is, the kinds of problems different methods are likely to find and what can be gained when they are combined. This paper will present findings from a meta-analysis of a selection of studies, carried out by the Questionnaire Development and Testing (QDT) Hub at the National Centre for Social Research, that have used a combination of different pre-testing methods. The meta-analysis will strive to focus on the kinds of data each method have provided and what this tells us about how such methods can be combined. Seven UK studies will be reviewed as part of the meta-analysis presented in this paper:
• an evaluation of a travel record used as part of the National Travel Survey involving coder debriefings, secondary analysis of data, cognitive interviews, respondent debriefing, a field pilot and a comparison of errors made in a sample of completed existing travel diaries with those in a sample of completed ‘new’ diaries;
• An evaluation of a set of survey questions capturing attitudes towards violent extremism which involved respondent focus groups, an expert review and cognitive interviews;
• The development of a set of questions capturing gender, including transgender, identity involving focus groups with respondents, cognitive interviews and a small scale pilot;
• The evaluation of a set of questions asked within a disability module around choice and control which involved the application of the QAS(99) desk appraisal and cognitive interviews;
• The development of a set of questions around the provision and receipt of social care amongst people aged 65 and over involving respondent focus groups, an expert panel and two rounds of cognitive interviews;
• A review of questions around benefit receipt, looking at discrepancies between administrative data held about individuals and survey reported data, involving interviewer focus groups and cognitive interviews; and,
• The evaluation of a question used to collect respondent’s email addresses for use in follow up research which involved cognitive interviews, a pilot of the question and a validation exercise to assess the quality of the information collected.