2018 Annual Conference of China English Language Education Association (CELEA2018)

187 / 2018-07-05 11:07:50

The Validation of A Language Performance test with A Multi-dimensional Construct

language performance test,test validation,AUA,ETIC,MDC

Plenary track > 中国特色英语测评研究

Language performance tests based on a multi-dimensional construct (MDC) have been a new practice to meet the trends of fostering international talents who are not only proficient in language, but also are equipped with global vision, critical thinking ability and intercultural communication competence. Considering the complexity of the construct, the validation of such a test would be of primary concern. The emphasis of the validation is on the meaningfulness of the interpretations about the abilities to be assessed. Whereas few studies has been conducted on the issue concerned, this thesis aims at validating a MDC language performance test under the guidance of AUA (Assessment Use Argument) framework. The test under study is English Test for International Communication (ETIC), a newly developed large-scale performance test for college students in China. A mixed-method design has been employed to collect evidence both from the rating process and the rating outcomes to answer the central question of whether the construct of ETIC is effectively realized. Quantitatively, by analyzing the rating outcomes which includes 1000 test takers’ scores on two test tasks via Many-facet Rasch Model (MFRM) , this study is intended to figure out (1) if there exists rater bias on different categories of the rating scale; and (2) how well the raters spread test takers into different levels on all aspects of the construct based on the rating scale. Qualitatively, through think-aloud protocol and retrospective interview o 4 raters, the current research expects to identify whether the raters pay attention to all aspects of the construct and how the raters distribute their attention to the features of the concerned abilities. The results will be expected to, on one hand, shed light on the approaches to validating MDC language performance tests; and on the other, have some implications on improving the rating scale quality and the rater training effectiveness of high-stake performance tests.