Font Size:
Surveys of Legal Need Worldwide (and When Small Methodological Changes Can Result in Big Differences)
Nigel J Balmer, Pascoe Pleasence
Building: Holme Building
Room: Cullen Room
Date: 2014-12-10 09:00 AM – 10:30 AM
Last modified: 2014-10-31
Abstract
Since the mid-1990s, at least 26 large-scale national surveys of the public’s experience of justiciable problems have been conducted in at least 15 separate jurisdictions, reflecting widespread legal aid reform activity. While the majority of these surveys take their structure from Genn’s Paths to Justice survey (1999), they vary significantly in length, scope, mode of administration, types of problems included, survey reference period, data structure, data analysis and question formulation.
This paper draws upon surveys from across the world, contrasting their methodologies and comparing their findings, as well as setting out the potential for bias as a consequence of methodological variation. The paper then presents findings from five online experiments testing the impact of various question formulations on problem prevalence, use of advice and formal processes. Specifically, the experiments test whether varying the reference period, describing problems as ‘legal’, offering detailed as opposed to simple problem descriptions and describing problems as ‘difficult to solve’ had an impact on reported prevalence of justiciable problems, and whether presenting lists as opposed to a series of individual questions had an impact on use of advice or processes.
The experiments demonstrated that modest differences in question formulation yield significantly different results. Specifically: alteration of survey reference period did not result in a proportional change in problem prevalence; introducing problems as ‘legal’ or ‘difficult to solve’ significantly reduced prevalence; and introducing use of advisors/processes as multiple questions rather than as lists significantly increased their reported use.
The risks involved in comparative analysis (and particularly in looking beyond methodology when attempting to explain jurisdictional variation) are discussed. The importance for future studies of understanding the impact of methodology, learning from the lessons of the past, making technical details transparent and data available are highlighted.
This paper draws upon surveys from across the world, contrasting their methodologies and comparing their findings, as well as setting out the potential for bias as a consequence of methodological variation. The paper then presents findings from five online experiments testing the impact of various question formulations on problem prevalence, use of advice and formal processes. Specifically, the experiments test whether varying the reference period, describing problems as ‘legal’, offering detailed as opposed to simple problem descriptions and describing problems as ‘difficult to solve’ had an impact on reported prevalence of justiciable problems, and whether presenting lists as opposed to a series of individual questions had an impact on use of advice or processes.
The experiments demonstrated that modest differences in question formulation yield significantly different results. Specifically: alteration of survey reference period did not result in a proportional change in problem prevalence; introducing problems as ‘legal’ or ‘difficult to solve’ significantly reduced prevalence; and introducing use of advisors/processes as multiple questions rather than as lists significantly increased their reported use.
The risks involved in comparative analysis (and particularly in looking beyond methodology when attempting to explain jurisdictional variation) are discussed. The importance for future studies of understanding the impact of methodology, learning from the lessons of the past, making technical details transparent and data available are highlighted.