Mobile telephones are ubiquitous. One of the most widely used data services worldwide (Kuntsche & Robert, 2009), most Australians use SMS daily (Mackay & Weidlich, 2009). This is an unprecedented opportunity for researchers to communicate with participants at any location, or time (Haller et al., 2006). However there is limited rigorous, methodical examination of its properties as a tool for psychological research (Cocco & Tuzzi, 2012; Tomlinson et al., 2009), in particular how it relates to other research modes as a multi-modal supplement, or substitute for other modes.
A major issue of note for researchers seeking to administer instruments using SMS is the instrument’s length. Though the once strict 160 character per message limit has been bypassed by support for stitching multiple SMS together, the length of an SMS is still salient to many users (Battestini, Setlur, & Sohn, 2010). In other modes instrument length, operationalised here as the number of items, can meaningfully impact participant engagement (Dillman, Smyth, & Christian, 2009) and in turn, data quality and response rates (i.e. Burchell & Marsh, 1992; Jepson et al. 2005; LaMar & Gale, 1982).
Over two studies, this paper explores how translation of self-report, likert-style questionnaires an SMS format impacts upon the psychometric properties of those instruments, and how the length of the questionnaires might be related to cross-mode measurement invariance. In study one, instrument length was systematically varied. Factor analysis of 1011 online completions of the 20-item Ruminative Thought Styles questionnaire (RTS; Brinker & Dozios, 2009) guided the construction of 5, 10, and 15 item short form variants. 417 undergraduate students were randomly assigned one of the four length variants, responding via paper survey or SMS.
In study two, different instruments of varying length were used, with a total of 911 participants responding online or via SMS of one of the instruments. We examined the 10-item negative axis of the PANAS (Watson, Clark, & Tellegen, 1988); 16-item Acceptance and Action Questionnaire (Hayes et al., 2004); 42-item Depression Anxiety Stress Scale (Lovibond, 1995); or 60 item PANAS-X (Watson & Clark, 1994).
Results demonstrated that, whilst some questionnaires were problematic, an instrument as long as sixty items can be administered by SMS, with comparable response rates, internal reliability, and factor structures to online administration. However, in instruments over ten items in length, mean responses tended to be higher, leading to lack of equivalence in terms of latent means and intercepts.