Response latency measurement and eye tracking are two relatively novel methods for diagnosing problematic survey questions. Response latency refers to the time it takes respondents to answer a question and research has shown that longer response times are associated with difficult and complex questions (e.g., Bassili 1996, Draisma & Dijkstra, 2004). Eye tracking records respondents’ eye movements while they complete a questionnaire and measures, for example, how long they fixate on specific parts of a question. Similar to response latencies, longer fixation times and larger numbers of fixations can indicate flawed questions (e.g., Lenzner et al., 2011). While the two methods are certainly useful tools in theoretical research (cf. Galesic et al., 2008; Lenzner et al., 2010), little is known about their utility in the practical pretesting context (e.g., in testing draft questions).
On the one hand, they have some advantages over other pretesting methods, such as cognitive interviewing or behavior coding. First, both are nonreactive and objective measures of behavior that are not affected by the researcher (and the ways in which he or she tests the questions) and the research context. Second, they are natural byproducts of the question answering process, and hence their validity neither depends on the respondents’ ability to verbally express themselves and to communicate their own thoughts nor on the interviewers’ or the researchers’ experience and expertise in evaluating questions. On the other hand, the interpretation of both response latencies and eye-tracking parameters is anything but straightforward: longer response latencies and longer fixation times may not necessarily suggest a problem but may also indicate an increasing interest in the question or a more conscientious response style.
This paper examines the potential of response latencies and eye tracking as objective methods for pretesting survey questions and compares their relative effectiveness in identifying flawed questions. In a Web survey experiment (N=44), both methods were simultaneously used to test 28 question pairs (each consisting of a poor and an improved question version). The presentation focuses on three main research questions examined in this experiment: (1) How effective and reliable are both methods in identifying the poor vs. improved question versions? (2) How are both methods ideally implemented in the survey pretesting context? (3) What are the implications of both methods with regard to the time and costs required?