The 5th Annual International Conference of the Asian Association for Language Assessment (AALA) (AALA 2018)

215 / 2018-04-12 14:04:28

Investigating Local Dependence in a Large-Scale English Proficiency Test

Local item dependence,testlets,subskills,item response theory

Plenary track > Standards in developing language assessments

The local independence of items in a test is a basic assumption of item response theory (IRT). This study investigated whether local dependence exists among items sharing a single stimulus in listening and reading tests that are included in a large-scale English proficiency test battery for Koreans. The listening and reading tests include two and five testlets, respectively; each of the testlets consists of one listening or reading passage and two items. The subskills measured by the item pairs are (a) main idea - detail, (b) main idea - inference, and (c) detail - inference. The chi-square index (Chen & Thissen, 1997) was used to show the degree of local dependence for the item pairs. A unidimensional two-parameter logistic model (2PL), the testlet response model (TRT), and the bi-factor model were compared in terms of model fit indices and item parameter estimates. We also examined how subskill item pairs affect the degree of local dependence. Test response data were collected through multiple pilot tests. The first pilot test was taken by 2,846 incoming freshmen at a large research university in Korea, while the second pilot test was taken by 615 test takers of various backgrounds, including high school students, college students, and students enrolled at test-prep institutes. Local dependence was found in three of the seven testlets in the first pilot test. Content analysis of the testlets revealed a tendency for main idea - detail and main idea - inference subskill item pairs to show local dependence. Lastly, although the difference between the three models was clear in the discrimination parameter estimates, the difference was not large in the difficulty parameter estimates. The model fit was better using TRT and the bi-factor model compared to 2PL, and the difference in model fit was statistically significant. These results point to the need for item writer training on avoiding local dependence when writing testlets, especially if a unidimensional IRT model is to be used to analyze the test results.