Nakatsuhara, Fumiyo

English

University of Bedfordshire Repository

CRELLAFumiyo NakatsuharaCRELLA, University of Bedfordshire, UKCentre for Research in English Language Learning and AssessmentCRELLA: its Socio-Cognitive Approach to Validating TestsXIX NATE, Surgut, May 15-19 2013 CRELLA (Centre for Research in English Language Learning and Assessment)www.beds.ac.uk/crella2 CRELLA Research StaffProf Cyril Weir: Director of CRELLAProf Stephen Bax: Professor in Applied LinguisticsDr Tony Green: Reader in LanguageAssessmentDr Vladimir Zegarac: Reader in Language and CommunicationDr John Field: Senior Lecturer in Cognition in Language Learning and AssessmentDr Lynda Taylor: Senior Lecturer in Language AssessmentDr Fumiyo Nakatsuhara: Senior Lecturer in Language AssessmentDr Chihiro Inoue: Post-doctoral Research FellowProf Liz Hamp-Lyons: Visiting ProfessorProf Roger Hawkey: Visiting ProfessorRebecca van der Westhuizen: Research AdministratorOverview1. Background2. Weir’s (2005) Socio-cognitive framework for test validation3. Example validation studies drawing upon the Socio-cognitive framework 4. ConclusionCRELLA1. BackgroundCEFR: usefulness and limitationsCommon European Framework of Reference for Languages: Learning, teaching, assessment (CEFR; Council of Europe, 2001)• Useful for learners’ needs analysis, syllabus/curriculum designs, provision of feedback to learners, comparison between different tests• The CEFR is deliberately underspecified should be seen as a heuristic device.• NOT all tests linked to the CEFR are satisfactory from a quality perspective. (Alderson, 2004; Fulcher, 2004, Green 2012; Milanovic & Weir, 2010; North, 2000; O’Sullivan & Weir, 2011; Weir, 2005b)Need for a validation framework•to consider and incorporate criterial contextual, cognitive and evaluative (scoring) parameters at the test development stage •to guide us in generating evidence of the successful operationalisation of these features at the test implementation stage7• Language testers’ responsibility: To provide test stakeholders with information about what the test construct is and how they are operationalised.CRELLA2. Weir’s (2005) socio-cognitive framework for test validationCONTEXT  VALIDITY COGNITIVE VALIDITYResponseSCORING VALIDITYTest-taker characteristicsScore / GradeCONSEQUENTIAL VALIDITYCRITERION-RELATED VALIDITYSocio-cognitive framework•The framework represents a unified approach to gathering validation evidence for a test, and shows how the various validity components fit together both temporally and conceptually.•The timeline runs from top to bottom, offering test developers a plan of validation studies.•Now used by test providers including:– The British Council (IELTS, ILA, Aptis)– Cambridge English Language Assessment (KET, PET, FCE, CAE, CPE)– Eiken Foundation of Japan (EIKEN, TEAP)– The Language Training and Testing Center, Taiwan (GEPT)– Trinity College London (ISE, GESE)10Critical questions to be addressed1. [Test taker characteristics] How are the physical/physiological, psychological and experiential characteristics of candidates catered for by the test? 2. [Cognitive validity] Are the cognitive processes required to complete the test tasks appropriate? 3. [Context validity] Are the characteristics of the test tasks and their administration appropriate? 4. [Scoring validity] How far can we depend on the scores, which result from the test?  5. [Consequential validity] What effects do the test and test scores have on various stakeholders?6. [Criterion-related validity] What external evidence is there that the test is measuring the construct of interest?11CRELLA3. Example validation studies drawing upon the Socio-cognitive frameworkCognitive validityResearch on onscreen reading tests with eye tracking technologyBax and Weir (2012); Bax (in press)(Funded by the British Council, Cambridge English Language Assessment)Levels of cognitive processing in reading tests (Bax, in press, adapted from Khalifa & Weir, 2009)Cognitive Processes Size of typical reading unitWord recognition: Word matching WordLexical access: Synonym and word class matchingWordGrammatical parsing Clause/SentenceEstablishing propositional meaning SentenceInferencing Sentence/Paragraph/TextBuilding a mental model TextUnderstanding text function TextContext validity18A study of examiner interventions in relation to the listening demands they make on candidates in oral interview testsNakatsuhara & Field (2012)(Funded by Trinity College London)Socio-cognitive framework: Contextual  parameters for Speaking Tests(Weir, 2005a; Taylor ed. 2011)SETTING: TASK•Response format•Purpose •Weighting •Knowledge of criteria•Order of items/tasks•Time constraintsSETTING: ADMINISTRATION•Physical conditions•Uniformity of administration•SecurityDEMANDS: TASKLinguistic (Input and Output)•Channel•Discourse mode•Length•Nature of information•Topic familiarity / content knowledge•Lexical resources•Structural resources•Functional resourcesInterlocutor•Speech rate•Variety of accent•Acquaintanceship•Number•Gender 19The role of listening in interactive speaking tests• Oral interview tests are to some extent tapping into the construct of listening-into-speaking, i.e. interactive listening skills (e.g. Nakatsuhara, 2012)• Trinity’s GESE (Graded Examinations in Spoken English) exams: assessing both speaking and interactive listening skills through communicative interaction. Research QuestionWhat types of examiner intervention are employed in the GESE examinations in terms of their linguistic and discourse features?20MethodsStage 1: Transcribe audio-recorded test sessions obtained by Trinity Stage 2: Select contextual parameters in the spoken input and analyse the examiner interventions for:1) Lexical complexity2) Syntactic complexity3) Informational density4) Number and mean length of interventions5) Speech rate6) Purpose for interventions21CRELLA5. Conclusion• The Socio-cognitive framework identifies the evidence required to develop a transparent and coherent validity argument.• The framework is theoretically sound yet operationally useful when we develop and validate tests.For more information:• Geranpayeh, A. & Taylor, L. (eds.) (2013). Examining Listening, Cambridge: CUP.• Khalifa, H. & Weir, C.J. (2009) Examining Reading, Cambridge: CUP.• Shaw, S.D. & Weir, C. J. (2007) Examining Writing, Cambridge: CUP.• Taylor, L. (ed.) (2011) Examining Speaking, Cambridge: CUP.• Weir, C. J. (2005) Language Testing and Validation: an Evidence-Based Approach, Basingstoke: Palgrave Macmillan. CRELLAThank you!Fumiyo NakatsuharaCRELLA, University of Bedfordshire, UKfumiyo.nakatsuhara@beds.ac.ukXIX NATE, Surgut, May 15-19 2013 

CRELLA: its socio-cognitive approach to validating tests

http://uobrep.openrepository.com/uobrep/bitstream/10547/623694/2/nate-nakatsuhara-18-may-2013-1.pdf

CRELLA: its socio-cognitive approach to validating tests

Abstract

Similar works

Full text

Available Versions

University of Bedfordshire Repository