ACSPRI Conferences, RC33 Eighth International Conference on Social Science Methodology

Font Size:  Small  Medium  Large

Detecting conditions of high versus low validity and reliability using unobtrusive measures in surveys.

Jochen Mayerl, Piet Sellke

Building: Law Building
Room: Breakout 11 - Law Building, Room 107
Date: 2012-07-12 01:30 PM – 03:00 PM
Last modified: 2011-12-08


Data quality in standardized surveys is a core issue for decades in survey research. Extensive pretesting and post-survey data treatment is required in order to minimize measurement errors. However, most methods seem to be rather coarse. Behavior coding and measurement of response latencies, however, are two promising unobtrusive methods which can be used to gain a better understanding of respondents’ behavior and thus to enhance the data quality of computer-assisted surveys. Using both methods to detect measurement errors makes up a new differentiation, leading to a better understanding of cognitive processes.

The paper shows that behavior coding and response latencies can be used to identify specific types of cognitive response modes. The key argument is that these response modes moderate the occurrence of response effects, measurement errors and thus the reliability and validity of attitudinal measurement models. Additionally, the paper discusses correlates of behavior coding which could be used as proxy variables to adopt behavior coding results to large-scale surveys.

Data of a two-wave German CATI-survey with 2002 respondents are used to compare results of an application of behavior coding and response latency measurement to analyse data quality of a measurement model of attitudes towards health nutrition. Response effects are analysed by experimental variation of question order of negative and positive worded items. Structural equation models are estimated in a multiple-group moderator design to test validity and reliability of the latent attitude construct.