Several sections of the World Values Survey (WVS) were administered in parallel on Life in Australia™, Australia’s only probability-based online research panel, which includes offline respondents via telephone surveys. The parallel administration was used to conduct a number of methodological experiments.

Experiment 1 compared grids and item-by-item administration for web respondents. Grids are broken into single items on mobile devices. A consistent concern is that the different presentation format will impact inter-item correlation in scales. Desktop respondents were randomised into grid and item-by-item displays. There was no significant difference in coefficient alpha by display format.

Experiment 2 compared wide and long presentation for 10-point semantic differential scales. The wide scales used on WVS cannot be effectively displayed on mobile devices. Desktop respondents were randomised into long and wide conditions. There were no significant differences in median response or variance by display format.

Experiment 3 tested whether recency bias in telephone-administered surveys could be offset by reversing scale order. Telephone respondents were randomised into standard and reversed response order. No recency bias was found for Likert scales. Significant effects were, however, found for a more cognitively challenging item that asked respondents to select up to five options from a bank.

Experiment 4 tested whether recency bias between telephone-administered and web surveys could be offset by reversing scale order. As mode was not randomised, coarsened exact matching was used to adjust for non-ignorable differences in mode of response. Inconsistent effects mode effects were observed: the null hypothesis was rejected in 2 of 7 tests.

Given the unexpected results observed in Experiment 4, additional tests were conducted for acquiescence response bias and social desirability bias using items that had not been analysed. In the majority of cases, acquiescence response bias was observed: the null hypothesis that telephone mode would differ from web was rejected in 4 of 6 tests. Inconsistent effects for social desirability bias were observed: in 0 of 4 cases where social desirability was expected were significant effects observed in the expected direction; however, in 2 of 4 cases where no social desirability effects were expected, significant differences between modes were observed.