The 5th Annual International Conference of the Asian Association for Language Assessment (AALA) (AALA 2018)

170 / 2018-03-28 07:08:38

Construct validation of a receptive test of collocational knowledge: Complementary use of CTT and many-facet Rasch measurement

collocations,corpus-based,argument-based approach,validation

Plenary track > Standards in validating language assessments

Collocations are essential, ubiquitous and naturally-occurring in both written and spoken English (e.g. Erman & Warren, 2000; Sinclair, 1991). Thus, when used incorrectly, collocations negatively affect fluency and mark even high-proficiency learners as non-native speakers, even if they produce grammatically correct utterances (e.g. Shin & Nation, 2008). As no validated test exists for collocations (Webb and Sasao, 2013), the current research aimed to create a corpus-driven receptive test of English collocational knowledge by considering how properties of collocations (listed below) affect item difficulty. 132 non-English-dominant and 47 English-dominant undergraduates at an Australian university took the pilot test, while 261 non-English-dominant students studying either in preparation for or at an Australian university took the revised test (along with 29 English teachers).

This paper will focus on the complementary use of Classical Test Theory (CTT) and many-facet Rasch measurement (MFRM) in construct validation using data from the pilot and revised tests. Initially, CTT was used for the identification and subsequent removal or revision of problematic items. Due to the large number of potentially relevant collocation properties in addition to two criterion measures (knowledge of routine formulae and general proficiency), the number of participants was insufficient for all the properties to be included in the MFRM analysis. Thus, step-wise regressions were first used to identify those which most predicted item difficulty, leaving only type (five subtests: compound words and four collocation types), frequency, degree of coherence, and semantic transparency for the analysis. While a few of these could significantly predict item difficulty, it was collocation type that did this best. As expected, both lexical collocations (verb-noun, adjective-noun) were significantly easier than the grammatical ones (noun-preposition, other phrase), mostly due to differences in semantic transparency.

The data suggest that the construct may not be limited to collocational knowledge but to a larger construct of vocabulary knowledge. For instance, internal consistency was high with the compound word subtest (α=.98) and without (α=.97). The internal consistency of all subtests after the deletion of problematic items (15-18 collocations each) was reasonably high (α between .72 and .91). MFRM analysis produced similar findings and provided a clearer picture of the comparable effects of collocation properties. In sum, this paper provides strong support for the inclusion of multiple collocation types ranging in semantic transparency and frequency. This careful item selection process is an important and often neglected aspect of the development and validation of collocation tests.