Font Size:
The Online Panels Benchmarking Study: A comparison of the relative accuracy of population estimates obtained from probability surveys and non-probability online panels.
Darren Walter Pennay, Dina Neiger, Paul John Lavrakas, Nikki Honey, Kim A Borg
Building: Holme Building
Room: Holme Room
Date: 2016-07-21 11:00 AM – 12:30 PM
Last modified: 2016-05-06
Abstract
In Australia in 2013, 83 per cent of households had the internet connected with 77 per cent having a broadband connection (ABS Cat.8146.0). In 2013 online research accounted for 34 per cent of the revenue generated by the Australian market and social research industry (28% worldwide) up from 29 per cent two years earlier (ESOMAR, 2014).
There are approximately 15 national online panels operating in Australia, the oldest of which were established in the late 1990s. While some of the commercial panels operating in Australia mention offline recruitment as one of the methods used to recruit members, to our knowledge there is no commercially available national probability-based online panel operating in Australia.
For several years now, the authors of this paper have been concerned that the rapid increase in the use of non-probability online panels in Australia has not been accompanied by an informed debate regarding the advantages and disadvantages of using such panels.
To inform this debate the Social Research Centre undertook the 2015 Australian Online Panels Benchmarking Study. In this paper we will report on the findings from eight national surveys – three probability surveys and five surveys conducted via non-probability online panels. The three probability surveys consist of a high-effort RDD dual-frame CATI survey, a dual-frame RDD CATI Recruit-to-Online survey, and a probability survey using an address based sampling frame.
To our knowledge this is the first Australian study to compare the accuracy of the results obtained from surveys administered using probability-based sampling methods with those obtained from surveys administered to members of independent online panels recruited using non-probability methods, with virtually the same questionnaire used in all the studies. This will enable us to determine whether the findings from other international studies which have shown that surveys using probability sampling methods produce more accurate results than surveys relying upon non-probability sampling methods. (e.g. Yeager et al. 2011, Chang & Krosnick 2009, Walker, Pettit & Rubinson, 2009) hold true in an Australian context. The results of this study have implication for the establishment of a probability-based online panel in Australia.
There are approximately 15 national online panels operating in Australia, the oldest of which were established in the late 1990s. While some of the commercial panels operating in Australia mention offline recruitment as one of the methods used to recruit members, to our knowledge there is no commercially available national probability-based online panel operating in Australia.
For several years now, the authors of this paper have been concerned that the rapid increase in the use of non-probability online panels in Australia has not been accompanied by an informed debate regarding the advantages and disadvantages of using such panels.
To inform this debate the Social Research Centre undertook the 2015 Australian Online Panels Benchmarking Study. In this paper we will report on the findings from eight national surveys – three probability surveys and five surveys conducted via non-probability online panels. The three probability surveys consist of a high-effort RDD dual-frame CATI survey, a dual-frame RDD CATI Recruit-to-Online survey, and a probability survey using an address based sampling frame.
To our knowledge this is the first Australian study to compare the accuracy of the results obtained from surveys administered using probability-based sampling methods with those obtained from surveys administered to members of independent online panels recruited using non-probability methods, with virtually the same questionnaire used in all the studies. This will enable us to determine whether the findings from other international studies which have shown that surveys using probability sampling methods produce more accurate results than surveys relying upon non-probability sampling methods. (e.g. Yeager et al. 2011, Chang & Krosnick 2009, Walker, Pettit & Rubinson, 2009) hold true in an Australian context. The results of this study have implication for the establishment of a probability-based online panel in Australia.