2018 Annual Conference of China English Language Education Association (CELEA2018)

247 / 2018-06-29 09:05:53

Evaluating two rater standardization procedures of the ETIC test: Rater performance and perceptions

ETIC test,rater standardization,multifaceted Rasch analysis,rater performance

Plenary track > 中国特色英语测评研究

This study investigated the effectiveness of two parallel rater standardization procedures of the English Test for International Communication (ETIC). Twelve university English teachers participated in the study as trainee raters and were divided into two groups that underwent two modes of standardization led by the same trainer: collaborative and self-paced frame-of-reference training. Sample scripts from the email writing task of ETIC Intermediate were used as training materials. Raters in both groups first studied the same set of materials; the difference was that in the collaborative group raters discussed with both peers and the trainer while in the self-paced group raters only discussed their progress with the trainer. Raters then scored another set of 80 scripts so that their performance could be evaluated. Two multifaceted Rasch measurement models were used to examine rater performance, including rater severity, consistency, bias towards certain criteria and rating accuracy. In addition, two sessions of focus group interview were held to gauge raters’ perceptions of the training procedures. The study will contribute to the validity argument of the test and the current understanding of rater standardization processes in performance assessment.