Chan, Sathena Hiu Chong

Weir, Cyril J.

English

University of Bedfordshire Repository

Are current academic reading tests fit for purpose?Cyril Weir and Sathena Chan CRELLAConstruct construct constructAlan Davies wrote in the first issue of the journal Language Testing:“…in the end no empirical study can improve a test’s validity... What is most important is the preliminary thinking and the preliminary analysis as to the nature of the language learning we aim to capture.”Davies, A. (1984). Validating three tests of English language proficiency. Language Testing, 1 (1), 50-69.2The EAP reading constructSome examples of the empirical research data we drew on to establish the parameters that might be included in a model of reading for academic purposes: –Weir (1983) who investigated the language activities and associated problems of students studying at tertiary level throughout the UK in the early 1980’s; Responses were received from 940 overseas students, 530 British students and 560 academic staff, in respect of 43 post graduate courses, 61 undergraduate courses and 39 ‘A’ level courses–Weir, Yang and Jin (2000) who carried out a similar analysis of academic English needs of first year undergraduates in China; –Weir, Hawkey, Green, Unaldi and Devi (2007/12) who carried out a needs analysis of undergraduates’ and postgraduates’ language needs and problems at the University of Bedfordshire in the 21st century. 766 students responded.3Research Study 1 Weir (1983) on academic readingIn   summary, Weir’s (1983) study:•reveals the importance of assessing bothcareful and expeditious global reading ina test of academic English;•makes clear the need for students toprocess information at both the wholetext and intertextual levels, rather thanjust comprehending at the paragraph orsentence levels4Research Study 2 Weir, Huang and Yan’s (2000) study recommends:• Texts used in academic English tests should mirror the contextual variables found in real life academic texts • Expeditious reading must be tested as well as careful reading in academic English tests• Separability of reading skills should be mirrored in the way results are reported• If a choice is to be made in what is tested, global items i.e. comprehension beyond the sentence must always take precedence over local items.5Research Study 3For those intending to develop EAP reading tests, Weir et al’s (2012a) study provides clear evidence for the necessity of:• a dedicated expeditious global reading task to assess search reading of a longer text under time constraints• items testing the higher-level skills of understanding main ideas, • items testing how a text is structured• items testing the ability to integrate information across different texts  6“ACADEMIC READING” TESTSi) the contextual parameters in academic reading testsii) the types of reading required for measuring the construct of academic reading iii) the levels of processing in reading which should be prioritised in tests of academic reading, iv) formats and items in tests of academic readingv) assessing higher-level academic reading through an integrated task type, 7i)  Contextual parametersThere are a number of quantifiable differences between the texts that students are exposed to in their first year of study and those used in IELTS reading passages.IELTS texts do appear in some respects to be “easier” than the UG texts students are faced with. The main areas of concern in our data are that the IELTS texts:1. generally include a lower proportion of sub-technical academic vocabulary than the undergraduate texts 2. IELTS texts may not fully reflect the level of readability found among the more challenging academic texts that first year undergraduates might expect to encounter.  8Lexical/syntactic complexityLength C1000 words IELTS average 854:  min 589- max1063, CAE max 1100, GEPT ave 700, TOEFL ave 700 (Density and complexity of idea units important here)Type Token Ratio    (> 55% ) CAE   59. 95 TOEFL 55 5   GEPT 47.74 IELTS 53.68  UG 51.7Lexical density   (> 55% ) CAE   O.52    TOEFL 0.59   GEPT 0.59    IELTS ,O.57   UG 0.56Mean of modifiers per noun phrase (0.95-1.0) CAE   0.92  TOEFL 1.  GEPT 0.98   IELTS 0.98    UG 0.95Mean number of words before main verb in main clause (5.5) CAE   4.4      TOEFL 5.7       GEP T 5.52, IELTS 5.48    UG 4.59*Academic Word List >10% IELTS.7.9,  iBT TOEFL 7.84, CAE 4.58,  GEPT 7.0 , UG 10.5 %Grammatical level a range of sentence patterns, frequent compound sentences and many complex sentencesAverage sentence length 20-22 IELTS average 21.89, CAE 20.42, TOEFL 19.26, GEPT Advanced 21, UG 21.47*Readability Flesch-Kincaid Grade Level of =/>13.5 IELTS average FK 12.64 TOEFL 12.07 UG 13.5, GEPT 12 CAE 9.6910IELTS Undergraduate7.51012.51517.520Flesch-Kincaid Reading Grade LevelTest 8, Text 1The IELTS compromise: One size fits all?An issue to resolve is whether current or anyrevised academic reading tests can be usedappropriately in a variety of domains orwhether in the future, variants may need to beoffered for example in professional as againstacademic situations11Appropriate for professions? Read and Wette (2009) investigated the attitudes of a group of overseas-trained health professionals towards the IELTS test in their experiences of meeting the English language requirements for professional registration. Their study found that participants were generally favourably disposed towards the test being used for this purpose, while at the same time recognising that IELTS “... had limited capacity to assess their ability to communicate effectively in clinical settings” (3) Read and Wette note in relation to this response that IELTS is still designed primarily as a test for those entering educational and training contexts, and is not specifically intended “to assess the communication skills required in particular professions” (p 4). 12Appropriate for low level learners?• As well as concerns for its suitability for assessing medical staff and the professions, questions have also been raised, for example, about its suitability for assessing lower level EAP students in the Middle East. Morrow (2017:1) asks: • “While the IELTS exam has earned a reputation for its usefulness in many international educational contexts, it does not appear to fully address the specific needs and issues of students and universities in the United Arab Emirates... its influence on instructional and assessment practices has grown ... in ways that do not seem to be completely congruent with the needs of low-level English learners who are about to begin studies in English-medium universities.” 13Readiness to enter: time for a change?The concept of ‘readiness to enter’ has been used as an argumentfor current IELTS to be like it is i.e. general and not based squarely onthe academic domain or on professional practice, a test for allseasons as it were rather than fit for one particular purpose.Do we need a general test of a candidate’s baseline competence, atsay B2/C1 levels in the four skills which predicts readiness to enterprofessional or academic situations (rather like Aptis in fact) or do werequire a test of whether the candidate can actually cope with thedemands initially faced in a profession or in an academic context?If it is the former, little change is needed. If it is the latter, thenclearly more than one ‘authentic’ needs driven test will be requiredand the logistical challenges to be faced that much greater.14TOEFL: an  academic English testThe TOELF iBT, which is radically different from earlier versions of this test, clearly definines itself as a measure of academic language ability and aligns itself more closely to “the variety of language use tasks that examinees are expected to encounter in everyday academic life’ (Sawaki et al., 2009, p. 5).Sawaki, Y.; Stricker, L. J.; Oranje, A. H. (2009). Factor structure of the TOEFL Internet-based test. Language Testing, 26 (1), 5–30.15What is an academic reading test? He who loves practice without theoryis like the sailor who boards shipwithout a rudder and compass andnever knows where he may cast.Leonardo da Vinci The Notebooks of Leonardo Da Vinci, translated by Jean Paul Richter (1888), I - Prolegomena 16ii) Types of reading required17Expeditious reading‘He has only half learnt the art of readingwho has not added to it the even morerefined accomplishments of skipping andof skim-ming.’Arthur Balfour “The pleasures of reading” (1905: 32) 18Careful v Expeditious ReadingWeir et al (2013) point out that though a great deal ofattention has been paid to the assessment of slow,careful, incremental reading, expeditious reading (fast,efficient, selective reading) has not been explicitly testedin very many high stakes examinations to this day, despiteits inclusion in CEFR reading descriptors for academicpurposes at the C1 levels (Council of Europe, 2001).The 1991 - Cambridge CAE reading paper was a notableexception to this, as were the later multiple matchingtasks in the PET and FCE.19Separate expeditious paperThe Higher Intermediate General EnglishProficiency Test (GEPT) in Taiwan has had aseparate expeditious reading paper since itsinception in 2000 (Weir 2005a, Wu & Lin 2008,Wu 2013) but apart from GEPT and the Test forEnglish Majors (TEM) in China (See Zou, Green &Weir, 1997) these appear to be the only highstakes English language examinations whichhave papers dedicated to expeditious reading.20Need an expeditious reading task• Weir et al (2012b: 179) conclude that there should be a dedicated section in any academic reading test to assess expeditious global reading. Such a test would impose explicit time constraints that encourage the use of the expeditious global reading strategies essential for university study. • If the test is computerised then such an expeditious task is clearly both possible and desirable. • Such dedicated tasks are likely to have beneficial washback if they equip students with the strategies to read texts selectively, quickly and efficiently to establish global meaning, as they have to do in real life reading activities in an academic context.2122Remediation wherenecessaryMonitor:goal checkingGoal setterSelecting appropriatetype of reading:Careful readingLOCAL:Understanding sentenceGLOBAL:Comprehend main idea(s)Comprehend overall textComprehend overall textsExpeditious readingLOCAL:Scan/search for specificsGLOBAL:Skim for gistSearch for main ideas andand important detailVisual inputGeneral knowledgeof the worldTopic knowledgeMeaning representation of text(s) so farSyntactic knowledgeLexiconLemma:MeaningWord classLexiconForm:OrthographyPhonologyMorphologyBuilding a mental modelIntegrating new informationEnriching the propositionInferencingEstablishingpropositional meaningat clause and sentence levelsSyntactic ParsingLexical accessWord recognitionText structure knowledge:Genre Rhetorical tasksCreating a text level representation:Construct an organised representation of a single textCreating an intertextual representation:Construct an organised representation across textsiii) The levels of processing in reading which should be prioritised in tests of academic readingThe research literature on reading in anacademic context, and on IELTS reading inparticular (see for example Moore, Morton andPrice, (2007), Green and Hawkey, (2012), Weir etal, (2012a and 2012b), Chalmers andWalkinshaw (2014, p.24), Bax (2015), and Owen(2016), are unanimous that academic readingtests should involve higher order readingprocesses rather than lower order processes.23Owen 2016 Doctoral ThesisOwen concludes in his findings on IELTS and TOEFL (2016: 366-367):The majority of identified cognitive processes were lower levelprocesses for both tests. … high level processing in both tests liesprimarily in ‘forming a mental model’ (P6) rather thaninferencing or creating a text-level representation… Some of thehigher-level processes in Khalifa and Weir’s model are under-represented in both IELTS and TOEFL, including inferentialreasoning and forming a text-level representation24Global versus local reading• By the twenty first century the IELTS reading test appears to be for themost part a test of general reading ability with items focusing mainly onthe local level.• If we examine the cognitive focuses of the current IELTS academicreading we can see how far the shift from global to local reading itemshas gone.• Given the clear need for first year university students to understand themain ideas in a text and how these relate to each other within andacross texts in writing their assignments, this preponderance of itemsfocusing on lower level reading skills must be a cause for some concern.• There is a clear need to ensure that future IELTS tests at the global levelinclude items that require creation of (i) a text level representation and(ii) an intertextual representation.25iv) formats and items in tests of academic reading• The prevalence of items testing at the lower levels of reading in current IELTS is indicative that a historical shift in the focuses of the test has occurred, a direction compounded by the item formats currently used.• Serious thought should be given to reducing the number of items/tasks that appear to initiate only lower order processing to ensure IELTS is only testing global meaning beyond the sentence; a sine qua non of academic reading.• The current formats testing lower order reading processes need to be replaced by item types that are likely to require higher order reading processes particularly item types  that lead to a textual or inter-textual representation.26Alderson on the fallacy underlying current reading testsHaving described the most common formats used for testingreading comprehension in great detail over c.50 pages (Alderson,2000, p. 202-248), Alderson then raises the uncomfortablequestion of whether any of the formats he has suggested couldbe said to represent what we actually do in reading in real lifei.e.:“correspond to the way they normally read such texts…the test may notreflect how students would understand the text in the real world…theonly purpose we typically give students for their reading is to answer ourquestions, to demonstrate their understanding or lack of it… readers donot usually answer somebody else’s questions: they generate and answertheir own” (Alderson 2000: 249)27v Assessing higher-level academic reading through an integrated task type29The thinness of the soil is often displayed by theEnglish Essay and frequently its prognosticationproves correct… (276)I would allow candidates while writing their essay inthe examination room to have access to somestandard authorities on their subject… there is nowno object in forcing men to carry a number of detailsin their heads…the range of subjects which can begiven for essays is very much extendedHenry Latham On the Action of ExaminationsConsidered as a Means of Selection 1877Beyond reading for comprehension– Critical thinking • synthesize multiple ideas into a theory – Reading• determine major and subordinate ideas in passages• summarize information• synthesize information from reading and incorporate it into a writing assignment• read texts of complexity without instruction and guidance• retain information while searching for answers to self-generated questionsIntersegmental Committee of the Academic Senates of the California Community Colleges and the University of California (2002) Academic Literacy: A Statement of Competencies Expected of Students Entering California's Public Colleges and Universities. Available at www. http://icas-ca.org/acadlit. 30Which academic reading skills can be tested better by the R/W format?In our view, three of the 14 academic reading skills identified in Weir (1983) could be tested better by the integrated format.•Separating the essential from the non-essential in a text•Note-making:(a) Extracting salient points for summary (b) Selective extraction of relevant points from a text (c) Reducing a text through rejection of redundant information•Critical evaluation: assessing the worth of a text and the way information in it has been organised and expressed31Evidence from the literature• When students are required to read for writing, they engage in the  more complex, higher order processes of constructing textual and intertextual representations when compared to responding simply to conventional reading comprehension items (Weigle, Yang, and Montee, 2013) • When reading to integrate materials, students construct elaborate models of the text structure and situation, enabling them to select information from the source text, evaluate it, and use it for writing purposes (Kintsch, 1998).• The various goals of reading-into-writing elicit different higher-level reading strategies, e.g. more integrative and analytic tasks leading to richer representation of the texts (Langer, 1980, 1986; Newell, 1984)..  • Reading ability was found to be linked to  characteristics of students’ syntheses of sources, especially organisation (Spivey, 1988). 32The advantages of using the integrated format to test academic reading skillsA reading-into-writing task• gives students a reader’s perspective (a goal for comprehending with a communicative intent of its own) – which in turn affects the attention allocated to different parts of the texts (Goetz, Schallert, Reynolds, & Radin, 1983; Just & Carpenter, 1980) • allows/encourages students’ autonomy to handle source materials rather than responding to pre-constructed representations• elicits deeper and more academically authentic reading processes as students interpret the full significance of the source material (Weigle, Yand and Montee, 2013)33Reading into writing• Reading into writing tasks represent the most situationallyand interactionally authentic test task for testing academicreading ability (and writing) i.e. they are the most constructvalid tasks for this purpose.• It may well be time to revisit the use of integrated readinginto writing tasks to better represent the higher orderprocessing in reading that is the hallmark of academic reading(and higher order processing in writing).• It is perfectly plausible to score the reading element of suchtasks according to the relevance and adequacy of the contentproduced, the way connections have been made betweenideas and organised, though the availability of a constructvalid piece of writing would be an added bonus and bringother criteria into play if it was to be included as part of thewriting assessment34Are current academic reading tests fit for purpose?35Can we develop fit for purpose academic reading tests?36

Are current academic reading tests fit for purpose?

http://uobrep.openrepository.com/uobrep/bitstream/10547/623728/2/weir-and-chan-testing-academic-reading.pdf

Are current academic reading tests fit for purpose?

Abstract

Similar works

Full text

Available Versions

University of Bedfordshire Repository