Exploring the role of phraseological knowledge in foreign language reading

Foreign language (FL) knowledge has been shown to contribute significantly to FL reading performance. Studies have contrasted the contribution of FL vocabulary and syntactic knowledge, following a dichotomous view of these components, producing mixed results. Despite the increasingly recognized formulaic nature of language, the contribution made by phraseological knowledge to reading ability has not been investigated systematically. This study examines the impact of a broader construct definition of linguistic knowledge – which includes a phraseological component – in explaining variance in reading performances. Test scores of 418 learners of English as a foreign language (EFL) were modeled in a structural equation model, showing that a phraseological knowledge measure outperformed traditional syntactic and vocabulary measures in predicting reading comprehension variance. Additional insights into the role of phraseological knowledge were gained through verbal protocol analysis of 15 EFL learners answering reading comprehension items that targeted the understanding of phrasal expressions within written context. The findings hint at an underestimated, but critical, role of phraseological knowledge in FL reading, and are relevant to both the assessment and the teaching of EFL ability

Development and initial validation of a diagnostic computer-adaptive profiler of vocabulary knowledge

Vocabulary knowledge is key to the successful use of any language skill (Nation & Webb, 2011) and learning to map a particular meaning to an L2 form for a great number of words is therefore crucial for learners of a foreign language. Vocabulary assessments can play a facilitating role in this learning process, which is why there is now an abundance of assessment tools to measure lexical knowledge. However, few of these tests have undergone sophisticated validation, even after their release into the public domain. Although vocabulary tests are used in numerous pedagogical and research settings, there has been “relatively little progress in the development of new vocabulary tests” (Webb & Sasao, 2013, p. 263). Instead, conventionalized traditions are being reiterated without questioning them. This PhD project has set out to address this gap of an innovative measure of vocabulary knowledge by developing a new diagnostic computer-adaptive measure of form-meaning link knowledge: The Vocabulary Knowledge Profiler. The present test development project started from scratch by questioning the underlying assumptions and trying to make design decisions based not only on theoretical considerations but empirical evidence. In a series of studies, three major weaknesses of existing vocabulary tests were problematized: (1) selection of item formats, (2) sampling in terms of unit of counting, frequency bands and representativeness, and (3) the general lack of validation evidence and validation models. These issues were explored across four studies in this thesis to design a novel instrument and gather initial validation evidence for it along the way. The first set of studies presented in this thesis investigated the usefulness and informativeness of different item formats for vocabulary tests and found in a comparison of four different formats that all formats show considerable error in measurement but the MC format may be the most useful because of its systematicity in overestimating scores. The second set of studies found support for the adoption of the lemma as an appropriate counting unit and for a new approach to frequency banding that takes into account the relative importance of frequency bands in terms of the coverage they provide. Based on these foundation studies, test specifications were drawn up and an item bank was created, which was subjected to a large scale trial to admit functioning items to an item pool for creating a computer-adaptive test. A study was conducted to compare two different computer-adaptive algorithms for implementation in the test design, suggesting that a “floor first” design would generate more consistent and representative score profiles. For initial validation evidence, a final study was then conducted to relate scores from the finished test to that of a reading comprehension measure. The findings of the studies presented throughout the thesis are then synthesized to produce an initial version of a validation argument in the structure of Bachman and Palmer’s (2010) Assessment Use Argument to outline both the necessary areas for further research before the launch of the test as well as the collected validation evidence to date that builds a tentative argument that the Vocabulary Knowledge Profiler and of the diagnostic decisions that are made based on its results and use are beneficial to English as a foreign language (EFL) learners and EFL teachers for classroom learning and teaching

Development and initial validation of a diagnostic computer-adaptive profiler of vocabulary knowledge

Vocabulary knowledge is key to the successful use of any language skill (Nation & Webb, 2011) and learning to map a particular meaning to an L2 form for a great number of words is therefore crucial for learners of a foreign language. Vocabulary assessments can play a facilitating role in this learning process, which is why there is now an abundance of assessment tools to measure lexical knowledge. However, few of these tests have undergone sophisticated validation, even after their release into the public domain. Although vocabulary tests are used in numerous pedagogical and research settings, there has been “relatively little progress in the development of new vocabulary tests” (Webb & Sasao, 2013, p. 263). Instead, conventionalized traditions are being reiterated without questioning them. This PhD project has set out to address this gap of an innovative measure of vocabulary knowledge by developing a new diagnostic computer-adaptive measure of form-meaning link knowledge: The Vocabulary Knowledge Profiler. The present test development project started from scratch by questioning the underlying assumptions and trying to make design decisions based not only on theoretical considerations but empirical evidence. In a series of studies, three major weaknesses of existing vocabulary tests were problematized: (1) selection of item formats, (2) sampling in terms of unit of counting, frequency bands and representativeness, and (3) the general lack of validation evidence and validation models. These issues were explored across four studies in this thesis to design a novel instrument and gather initial validation evidence for it along the way. The first set of studies presented in this thesis investigated the usefulness and informativeness of different item formats for vocabulary tests and found in a comparison of four different formats that all formats show considerable error in measurement but the MC format may be the most useful because of its systematicity in overestimating scores. The second set of studies found support for the adoption of the lemma as an appropriate counting unit and for a new approach to frequency banding that takes into account the relative importance of frequency bands in terms of the coverage they provide. Based on these foundation studies, test specifications were drawn up and an item bank was created, which was subjected to a large scale trial to admit functioning items to an item pool for creating a computer-adaptive test. A study was conducted to compare two different computer-adaptive algorithms for implementation in the test design, suggesting that a “floor first” design would generate more consistent and representative score profiles. For initial validation evidence, a final study was then conducted to relate scores from the finished test to that of a reading comprehension measure. The findings of the studies presented throughout the thesis are then synthesized to produce an initial version of a validation argument in the structure of Bachman and Palmer’s (2010) Assessment Use Argument to outline both the necessary areas for further research before the launch of the test as well as the collected validation evidence to date that builds a tentative argument that the Vocabulary Knowledge Profiler and of the diagnostic decisions that are made based on its results and use are beneficial to English as a foreign language (EFL) learners and EFL teachers for classroom learning and teaching

Towards a comprehensive, empirical model of language assessment literacy across stakeholder groups:Developing the Language Assessment Literacy Survey

While scholars have proposed different models of language assessment literacy (LAL), these models have mostly comprised prescribed sets of components based on principles of good practice. As such, these models remain theoretical in nature, and represent the perspectives of language assessment researchers rather than stakeholders themselves. The project from which the current study is drawn was designed to address this issue through an empirical investigation of the LAL needs of different stakeholder groups. Central to this aim was the development of a rigorous and comprehensive survey which would illuminate the dimensionality of LAL and generate profiles of needs across these dimensions. This paper reports on the development of an instrument designed for this purpose: the Language Assessment Literacy Survey. We first describe the expert review and pretesting stages of survey development. Then we report on the results of an exploratory factor analysis based on data from a large-scale administration (N = 1086), where respondents from a range of stakeholder groups across the world judged the LAL needs of their peers. Finally, selected results from the large-scale administration are presented to illustrate the survey’s utility, specifically comparing the responses of language teachers, language testing/assessment developers and language testing/assessment researchers

Unknown Vocabulary Density and Reading Comprehension : Replicating Hu and Nation (2000)

Hu and Nation's (2000) study, which stipulated that second language (L2) readers need to be familiar with 98% of lexical items for adequate text comprehension, has become highly influential in L2 vocabulary research and pedagogy. However, the 98% critical threshold figure is based on findings from a research project in which a regression analysis was conducted with only 66 university students in New Zealand. The present study replicated Hu and Nation's research in a context different from a typical Western, educated, industrialized, rich, and democratic context with a sample of 104 Sri Lankan adult L2 learners in a nonacademic context. They each took a Vocabulary Levels Test and read one of five versions of two reading texts at different levels of density of unknown words before answering comprehension questions. The results of the original study could not be fully replicated