The increase costs associated with mounting high quality CATI surveys and the precipitous decline in response rates for such surveys has seen many in our industry turn to non-probability online panels. The advantages of these panels include reduced costs (relative to other modes of data collection), quick turnaround times, respondent convenience, reduced social desirability bias due to the self-completion mode of collection, the ability to target hard to reach populations, multimedia functionality and computerised questionnaires. However, what about the quality? This paper presents the international research and now some Australian research showing that surveys administered to members of non-probability panels are considerably more biased (generally more than double the bias) and more variable than surveys administered using probabilistic sampling methods. In fact, the estimates produced from some non-probability samples are sometimes dangerously inaccurate.
However, this paper doesn’t just shine a light on a problem but presents the latest Australian and international research exploring ways to reduce this bias and improve the inferences which can be made from non-probability panels. In so doing we look at techniques such as model-based design weights, sample blending and calibration. These methods are equally applicable to both market and social researchers.