Current practices in reporting limits and biases in scholarly literature: An exploratory content analysis of four disciplinary political science journals
Pierre-Olivier Bédard, Mathieu Ouimet
Building: Law Building
Room: Breakout 11 - Law Building, Room 107
Date: 2012-07-11 01:30 PM – 03:00 PM
Last modified: 2012-06-19
Abstract
The disclosure of the methodological and analytical principles guiding a study – and by the same token, its potential limits and biases – is largely recognized to be central to any scientific endeavor. As stressed by Ioannidis (2007:324), “discussions of limitations are essential for genuine scientific progress: they are useful for understanding a research finding, translating the importance of the potential errors involved, placing the current work in context, and ascribing a credibility level to it”. There is empirical evidence showing that presentation and discussion of limitations in scholarly articles are obviously not the norm.
While this is a problem facing every researcher, from all fields alike, little is known of the extent to which limitations and biases are made explicit in empirical social science published literature. Focusing specifically on political science scholarly research, we ask the following questions: To what extent do political scientists report on the limitations and potential biases in their publications? What are the most common limitations reported? Is there a correspondence between those declared and the limitations usually attributed to the empirical and analytical approach employed?
Objectives
i. To document the extent to which political scientists report limits and/or biases in scholarly publications.
ii. To identify the common textual formulations when reporting biases and limits
iii. To underscore the implications of these communicative practices both for the academic and the non-academic actors likely to use research in their professional practices.
Methods
We performed a systematic content analysis of all 2011 peer-reviewed articles from four top journals in political science from four countries (Canadian Journal of Political Science, Revue Française de Science Politique, American Journal of Political Sciencel, British Journal of Political Science). Only original empirical studies (qualitative, quantitative or mixed-method) were retrieved for coding. Extracted data includes research objectives and hypotheses, data collection, sampling methods, data analysis techniques, and the textual formulation of limitations.
Implications
As there seems to be a greater demand for research evidence by non-academic organizations (government ministries, state agencies, NGOs, lobbies, etc.), it seems natural to scrutinize the evidence to be transferred and applied. It can be argued that taking into account limitations and biases, inherent to any research study, is likely to lead to a more judicious use of research. Consequently, to improve to way research is used by academic and non-academic actors, it appears fundamental that researchers themselves operate some changes in their practice, when reporting, sharing and communicating results.
While this is a problem facing every researcher, from all fields alike, little is known of the extent to which limitations and biases are made explicit in empirical social science published literature. Focusing specifically on political science scholarly research, we ask the following questions: To what extent do political scientists report on the limitations and potential biases in their publications? What are the most common limitations reported? Is there a correspondence between those declared and the limitations usually attributed to the empirical and analytical approach employed?
Objectives
i. To document the extent to which political scientists report limits and/or biases in scholarly publications.
ii. To identify the common textual formulations when reporting biases and limits
iii. To underscore the implications of these communicative practices both for the academic and the non-academic actors likely to use research in their professional practices.
Methods
We performed a systematic content analysis of all 2011 peer-reviewed articles from four top journals in political science from four countries (Canadian Journal of Political Science, Revue Française de Science Politique, American Journal of Political Sciencel, British Journal of Political Science). Only original empirical studies (qualitative, quantitative or mixed-method) were retrieved for coding. Extracted data includes research objectives and hypotheses, data collection, sampling methods, data analysis techniques, and the textual formulation of limitations.
Implications
As there seems to be a greater demand for research evidence by non-academic organizations (government ministries, state agencies, NGOs, lobbies, etc.), it seems natural to scrutinize the evidence to be transferred and applied. It can be argued that taking into account limitations and biases, inherent to any research study, is likely to lead to a more judicious use of research. Consequently, to improve to way research is used by academic and non-academic actors, it appears fundamental that researchers themselves operate some changes in their practice, when reporting, sharing and communicating results.