ACSPRI Conferences, ACSPRI Social Science Methodology Conference 2016

Font Size:  Small  Medium  Large

Analysis of Differential Item Functioning in Social Science Research using the Rasch Model

Curt Hagquist, David Andrich

Building: Holme Building
Room: Cullen Room
Date: 2016-07-22 09:00 AM – 10:30 AM
Last modified: 2016-05-06

Abstract


Comparisons across genders, age, social class and other sample groups are frequently made in social science research using instruments with composite items as outcome variables. In order to ensure that the differences in outcomes are not reflecting differences in the functioning of the instruments, the measures and the items have to work in the same way for the different sample groups to be compared. Lack of invariance across sample groups is commonly called Differential Item Functioning (DIF). Although there are an increasing number of social science studies focusing on DIF, there are still many examples of studies conducted without sufficient examinations of the instruments that the analyses are based on. Recent methodological work has shown some significant progress of analysis of DIF: the use of analysis of variance (ANOVA) as a method to simultaneously detect different types of DIF, the need to distinguish between real and artificial DIF and the trade-off between reliability and validity. The purpose of the paper is to present and summarise these recent advances in the methodology for DIF-analysis. An illustrative example is used to demonstrate how DIF can be identified, quantified and potentially accounted for using the Rasch model. For this purpose, Swedish data from the Health Behavior in School-aged Children (HBSC) study were used. Analyses with the Rasch model were conducted of a composite measure of psychosomatic problems constructed by summation of the students’ responses on eight items. The procedures applied in the analysis of the current data demonstrate how DIF can be identified efficiently by ANOVA, and how the magnitude of DIF can be quantified by resolving items according to identifiable groups and using principles of test equating on the resolved items. The results confirm that the items showing DIF initially should not be resolved simultaneously but sequentially. The results of the analysis also show that the real DIF in some items do affect the person measurement. In the current example, the gender differences in psychosomatic problems diminished after the resolution of the DIF items. Resolving items with DIF is not justified if the source of the DIF is relevant for the content of the variable; then resolving DIF may negatively affect the validity of the instrument in assessing the variable. Generally, decisions on resolving items to deal with DIF should not be solely based on the outcomes from the DIF-analyses, but also rely on external information that would facilitate the understanding and interpretation of real DIF.