Font Size:
Item Positions Effects on Item Difficulty and Item Discrimination: A Case Study with Parallel Test Forms
Xiaoxun Sun
Building: Holme Building
Room: MacCallum Room
Date: 2018-12-13 03:30 PM – 05:00 PM
Last modified: 2018-10-17
Abstract
Item Position effects are of particular importance in large-scale assessments when the test design requires parallel forms constructed using the same items scrambled into different positions in the forms. Modelling the impact of item positions in the tests on item difficulty estimates and item discrimination becomes critical. If items function differently due to their positions within a test form, the estimation of candidates’ performance may be compromised.
In this investigation, we studied the impact of item positions on item difficulty and item discrimination based on data collected on English test forms across Grade 3 to Grade 10 from an international testing program conducted by ACER. For each grade, there are three test forms, A, B and C, with 40 to 45 multiple choice questions. Each test consists of two types of units, Reading Comprehension units and Language Convention units. When assembling test forms, the Language Convention units with items in Vocabulary, Grammar, Punctuation and Spelling were placed in the same positions across three forms in order to minimise the position effect. Reading Comprehension units were shuffled around, the easier units were kept towards the front end of the test form, and the harder ones towards the rear.
The tests were calibrated based on the Rasch Model. Preliminary results showed that for all grades, Reading Comprehension items become more difficult when they are in a later position in a test form. The correlations of item difficulty across three forms were very high (>0.95), and there is no significant difference in item discriminations for the same item among three forms even when they are in different positions in the forms. These results show that the design of English tests had worked for minimising position effects.
Further analysis will be conducted to evaluate whether the item position differences across test forms will lead to item functioning differently in subgroups. In future studies, we would like to extend this work to test position effects in the Language Convention units.
In this investigation, we studied the impact of item positions on item difficulty and item discrimination based on data collected on English test forms across Grade 3 to Grade 10 from an international testing program conducted by ACER. For each grade, there are three test forms, A, B and C, with 40 to 45 multiple choice questions. Each test consists of two types of units, Reading Comprehension units and Language Convention units. When assembling test forms, the Language Convention units with items in Vocabulary, Grammar, Punctuation and Spelling were placed in the same positions across three forms in order to minimise the position effect. Reading Comprehension units were shuffled around, the easier units were kept towards the front end of the test form, and the harder ones towards the rear.
The tests were calibrated based on the Rasch Model. Preliminary results showed that for all grades, Reading Comprehension items become more difficult when they are in a later position in a test form. The correlations of item difficulty across three forms were very high (>0.95), and there is no significant difference in item discriminations for the same item among three forms even when they are in different positions in the forms. These results show that the design of English tests had worked for minimising position effects.
Further analysis will be conducted to evaluate whether the item position differences across test forms will lead to item functioning differently in subgroups. In future studies, we would like to extend this work to test position effects in the Language Convention units.