Online panels are routinely used as a method for gathering survey data. The majority of online panels are established via nonprobability sampling methods. In nonprobability panels, essentially anyone aware of an open invitation can volunteer to join the panel, providing they meet eligibility requirements. The nonprobability nature of the sampling processes, self-selection biases and non-coverage due to the exclusion of the off-line population all limit the usefulness of the data collected through these means. This is particularly the case for research that aims to make conclusions that are generalisable to a larger population.

A well-designed representative sample selected using probability methods, after post-stratification weighting, will reliably generalise to the population of interest. However, probability samples can be too expensive and limited in sample size and may take too long to administer or be unable to target small or specific populations. The feasibility and advantages of blending probability and non-probability samples and using calibration weighting to reduce overall bias are well documented in the literature (e.g. DiSogra, Cobb, Chan, Dennis 2011 JSM) but not previously evaluated in the Australian survey research context.

To close this gap, the Social Research Centre, with the support of the ANU Centre for Social Research and Methods, conducted an Online Panels Benchmarking Study (OPBS). This study administered the same questionnaire across five non-probability online panels and three probability-based samples. Two of the three probability-based samples were administered online, enabling mode effects to be effectively excluded from the comparisons between probability and opt-in samples. The study included early adopters (EA) profile questions that have been previously shown to differentiate reliably between probability and opt-in samples and be effective in reducing bias when used as calibration benchmarks.

OPBS allows us, in the Australian context, to examine the size and robustness of the differences between probability and opt-in respondents using questions that identify early adopters as well as to demonstrate the four step DiSogra approach (DiSogra et al 2011) to calibration weighting using these questions. By comparing estimates produced using DiSogra approach to independent benchmarks and estimates derived by other methods, we are also able to evaluate the potential of this approach for future Australian studies.