Chong , Sin Wang

Isaacs, Talia

English

St Andrews Research Repository

International Database of Education Systematic ReviewsRecord detailsIDESR IDIDESR000040TitleA protocol for a scoping review of learning-oriented language assessment in formal and non-formaleducational contextsMain Contact/Corresponding AuthorSin Wang Chong, Moray House School of Education and Sport, University of Edinburgh, UK; UCLCentre for Applied Linguistics, IOE-UCL's Faculty of Education and Society, University CollegeLondon, UK . sinwang.chong@ed.ac.ukHolyrood Rd, Edinburgh EH8 8AQ 220; Bedford Way, London WC1H 0ALAdditional AuthorsTalia Isaacs, UCL Centre for Applied Linguistics, IOE-UCL's Faculty of Education and Society,University College London, UK . talia.isaacs@ucl.ac.uk Review QuestionThe three research questions which guide this scoping review are: (1) How is learning-oriented language assessment conceptualised? (2) How is learning-oriented language assessment operationalised? (3) How is learning-oriented language assessment evaluated? RationaleLearning-oriented language assessment (LOLA) refers to the use of assessment to support learners'language development through the collection and interpretation of an array of evidence (Turner,2012). Such information provides insights into learners' current state of learning, their learninggoals, and ways to help learners progress towards their goals (Wiliam & Thompson, 2008). In otherwords, LOLA underscores the indispensable role of language assessment in bridging teaching andlearning (Colby-Kelly & Turner, 2007). LOLA, as an approach to language assessment, is grounded onformative assessment in educational assessment research (Black & Wiliam, 1998) and can takevarious forms, including classroom dialogues, peer assessment, self-assessment, and formative useof summative language tests (Turner & Purpura, 2016). The majority of the publications related toLOLA principles focus on classroom-based assessment (Fox et al., forthcoming; Lewkowicz & Leung,2021) but not assessment activities that take place beyond the classroom (Reinders & Benson,2017). With the rapid development of technology, LOLA is implemented in a wide range ofnon-formal language learning contexts including Language Massive Open Online Courses (MOOCs)/Small Private Online Courses (SPOCs) (Bárcena & Martín-Monje, 2014; Chong et al., 2022) andself-access language learning centres (Castellano et al., 2011). Focusing on LOLA in formal andnon-formal EFL contexts in higher education in Southeast Asia, the first author is leading a studycomprising a scoping review and a narrative inquiry, aiming to develop a literature-informed LOLAframework and examine English-as-a-Foreign-Language (EFL) teachers' and learners' experiences ofLOLA. This protocol pertains to the first phase of this study, that is, the scoping review. https://idesr.org Page 1/7International Database of Education Systematic ReviewsBárcena, E. & Martín-Monje, E. (2014). Introduction. LMOOCs: An emerging  field. In E. Martín-Monje & E. Bárcena (Eds.), LMOOCs: Providing learning, transcending boundaries(pp. 1-15). De Gruyter Open.  Black, P., & Wiliam, D. (1998). Assessment and classroom learning. Assessment in  Education: Principles, Policy & Practice, 5(1), 7-74. https://doi.org/10.1080/0969595980050102  Castellano, J., Mynard, J., & Rubesch, T. (2011). Student technology use in a self-access  center. Language Learning & Technology, 15(3), 12-27. http://dx.doi.org/10125/44256   Chong, S. W., Khan, M. A., & Reinders, H. (2022). A critical review of design features of  LMOOCs. Computer Assisted Language Learning. Advanced online publication. Colby-Kelly, C., & Turner, C. E. (2007). AFL research in the L2 classroom and evidence of usefulness:Taking formative assessment to the next level. The Canadian Modern Language Review, 64(1), 9-37.doi:10.3138/cmlr.64.1.009 Fox, J., Abdulhamid. N. & Turner, C.E. (Forthcoming).  Classroom based assessment.  In G.  Fulcher & L. Harding (Eds.), Routledge handbook of language testing (2nd ed.) (pp. xx-xx). Routledge.Lewkowicz, J., & Leung, C. (2021). Classroom-based assessment. Language  Teaching, 54(1), 47-57. https://doi.org/10.1017/S0261444820000506 Turner, C. E. (2012). Classroom assessment. In G. Fulcher & F. Davidson (Eds.), The  Routledge handbook of language testing (pp. 65-78). Routledge.  Turner, C. E., & Purpura, J. E. (2016). Learning-oriented assessment in second and foreign languageclassrooms. In D. Tsagari & J. Banerjee (Eds.), Handbook of second language assessment (pp.255-274). De Gruyter Mouton.  Wiliam, D., & Thompson, M. (2008). Interpreting assessment with instruction: What will it take tomake it work. In C. A. Dwyer (Ed.), The future of assessment: Shaping teaching and learning (pp.53-82). Lawrence Erlbaum Associates.  Inclusion CriteriaConceptual framework Include: There is a clear definition or conceptualisation of learning-oriented (language) assessment. Exclude: The term "learning-oriented (language) assessment" is used without a clear definition orconceptualisation.Rationale: RQ1 of this scoping review pertains to how LOLA is conceptualised. To address thisquestion, included studies need to define LOLA explicitly with reference to existing frameworks ordefinitions. LanguageInclude: Publications that are written in English.Exclude: Publications that are written in languages other than English. Rationale: Both authors are based in UK universities and their affiliations have access to mainlyacademic publications written in English. This is not to suggest that publications written in languagesother than English are less important; the consideration relates to accessibility. Type of publicationInclude: The publication belongs to one of the following types: journal articles (primary studies,research syntheses, commentaries, conceptual pieces), editorials, doctoral theses, books, bookchapters.Exclude: Publications that are descriptive; for example, purely reflective pieces or publications thatonly describe assessment practices are excluded.https://idesr.org Page 2/7International Database of Education Systematic ReviewsRationale: While descriptive and reflective pieces provide rich information about how LOLA isimplemented, and thus, answering RQ2 of this review, they are not likely to yield insights related tohow LOLA is conceptualised (RQ1) and evaluated (RQ3).   ContextInclude: In the case of a primary study, the context of the study is in higher education. Exclude: In the case of a primary study, the context of the study is not in higher education. Rationale: Learning-oriented assessment originates as an assessment concept in higher educationliterature and it is commonly practised in higher education institutes. Information SourcesSources of this scoping review will be selected from various online databases. Journal publicationswill be searched on Directory of Open Access Journals (DOAJ), ERIC, PsycINFO, Scopus, and SocialSciences Citation Index. Since LOLA is still an emerging area of research in language testing, therewill likely be a limited number of journal publications on this topic. Therefore, we consider includingother types of academic publications: doctoral theses, books, and book chapters. These will beretrieved from EBSCO, Electronic Theses Online Service (EThOS) (specifically for doctoral theses),and ProQuest. Search StrategyThe following search string will be used to conduct the scoping review:  ("learning-oriented assessment" OR "learning-oriented language assessment") AND (("highereducation" OR "universit*" OR "further education" OR "post-compulsory education" OR "vocationaleducation" OR "college" OR "undergraduate" OR "postgraduate") OR ("language education" OR"language*" OR "language teaching" OR "language learning" OR "TESOL" OR "ESL" OR "EFL" OR"English")) The search string was developed in consultation with two senior academics who have expertise inlanguage testing and language education and experience in conducting systematic literaturereviews. Additionally, the search string was reviewed by a subject librarian. The search string waspiloted together with the librarian in an online library module that the first author attended as a PhDstudent at UCL. The search string includes two major components: the assessment concept in focus(i.e., learning-oriented language assessment) and educational context (i.e., higher/further educationand language education). As for the assessment concept, we include both "learning-orientedlanguage assessment" and "learning-oriented assessment" because the latter is a more widely usedterm than the former. As for the educational setting, we attempt to capture synonymous or nearlysynonymous terms of "higher education", including the word "college", the US equivalent of"university". "College" is also used to refer to "university" in China. Additionally, to ensure that thepublications relate to language education, we include synonymous terms of "language education".Since the second phase of the study concerns learning-oriented language assessment in EFLcontexts, we also include key terms that locate publications on English language teaching andlearning. Data Management Publications that are deemed to be suitable for inclusion during first-level screening (screening oftitles and abstracts) will be imported, together with the full- texts in PDF format, to Zotero, areference management system. Second-level screening (screening of full texts) will be performed inZotero. Instead of removing excluded studies, they will be moved to a dedicated folder in Zotero forhttps://idesr.org Page 3/7International Database of Education Systematic Reviewsfurther review, if necessary. After the second-level screening and appraisal of study quality, the PDFsof the final set of studies will be imported to NVivo 12, a qualitative data analysis software for dataextraction and synthesis. Selection ProcessThe selection process, which comprises three stages: Identification, screening, and inclusion ofstudies, will be performed by the first author. In instances where there is uncertainty or ambiguity,the second author will be consulted. The process will follow and be represented in the 2020 versionof PRISMA diagram (Page et al., 2022). Potential publications will be identified using the aforementioned search string (see "SearchStrategy") on the databases mentioned in "Information Sources" by the first author in September2022, constantly referring to the inclusion criteria (see "Inclusion Criteria"). Following first-levelscreening, potentially relevant publications will be imported into Zotero and their full textsdownloaded as PDFs. The first author will then perform second-level screening, reading the wholetext to ensure all inclusion criteria are met. To ensure methodological rigour of the review process,an additional reviewer, who has experience conducting research synthesis, will independently screen20% of the publications included in first-level and second-level screening. The first author and thereviewer will meet to discuss and resolve discrepancies. Primary studies and research syntheses will be subject to an additional phase of appraisal todetermine their methodological rigour. Qualitative, quantitative, and mixed-methods research will beevaluated using the appraisal checklist of qualitative, quantitative, and mixed-methods researchpublished by TESOL Quarterly (2022), one of the flagship journals in language education. Synthesesof qualitative evidence will be evaluated following the methodological framework by Chong andPlonsky (2021) while appraisal of meta-analyses will be informed by Oswald & Plonsky (2010). Chong, S. W., & Plonsky, L. (2021). A primer on qualitative research synthesis in TESOL. TESOL Quarterly, 55(3), 1024-1034. https://doi.org/10.1002/tesq.3030Oswald, F. L., & Plonsky, L. (2010). Met-analysis in second language research: Choices and challenges. Annual Review of Applied Linguistics, 30(March, 2010), 85-110.https://doi.org/10.1017/S0267190510000115Page, M.  J., McKenzie, J. E., Bossuyt, P. M., Boutron, I., Hoffmann, T. C., Mulrow, C. D., et al. (2021). The PRISMA 2020 statement: an updated guideline for reporting systematic reviews BMJ,372, n71. https://doi.org/10.1136/bmj.n71TESOL Quarterly. (2022). TESOL Quarterly submission guidelines. Retrieved from https://onlinelibrary.wiley.com/page/journal/15457249/homepage/forauthors.htmlData Collection ProcessThe present review is a scoping review. A scoping review refers to a comprehensive, systematicliterature review on emerging substantive topics (Tricco et al., 2016). Comparing with other types ofresearch synthesis, for example, systematic literature reviews, scoping reviews usually adopt a moreinclusive approach to study selection, and they are more exploratory in nature (e.g., Chong &Reinders, 2022). The more encompassing nature of scoping review is reflected in the design of the data extractionform items. To ensure that the data extraction form is validated, an existing data extraction form forconducting a scoping review on a topic in the field of language education (Chong & Reinders, 2022)https://idesr.org Page 4/7International Database of Education Systematic Reviewswill be modified. Although this data extraction form was designed to survey the landscape of anothersubstantive topic, the overarching research questions of Chong and Reinders (2022) are identical tothe three in this review, focusing on conceptualisation, operationalisation, and evaluation. The formwas originally developed by the first author for Chong and Reinders (2022), reviewed andcommented by two senior academics in language education who have experience in conductingresearch syntheses. A modified version of this data extraction form will be used to extractinformation from the included studies for the present review. This modified data extraction form willbe reviewed by the second author, who is an expert in Language Testing and in conductingsystematic literature reviews. Once feedback from the second author is incorporated, the dataextraction form will be piloted on approximately 10% of the included publications, covering journalarticles, books/book chapters, and doctoral theses, by the first author and an additional reviewer.Any further changes will be made after the piloting phase. This final version of data extraction formwill be used to extract information from the remaining publications, again performed by the firstauthor and the additional reviewer. Chong, S. W., & Reinders, H. (2022). Autonomy of English language learners: A scoping review of research and practice. Language Teaching Research. Advanced online publication.https://doi.org/10.1177%2F13621688221075812  Tricco, A. C., Lillie, E., Zarin, W., O'Brien, K., Colquhoun, H., Kastner, M., … Straus, S. E. (2016). A scoping review on the conduct and reporting of scoping reviews. BMC Medical ResearchMethodology, 16(1). https://10.1186/s12874-016-0116-4 Data ItemsThe following items are based on the data extraction form of Chong and Reinders (2020) and aretentative. The items may be revised following the validation process mentioned in "Data CollectionProcess".Characteristics of included studies: • Type of publication • Year of publication • Location of authors • Location of study (only applicable to primary studies) • Research questions • Research design (qualitative, quantitative, mixed)Research question 1: Conceptualisation of learning-oriented language assessment • Conceptual framework  • Theoretical underpinningsResearch question 2: Operationalisation of learning-oriented language assessment • Context of the intervention  • Description of the intervention • Duration of the intervention • Materials used • Information about teachers • Information about learners • Role of the teacher • Role of the learnerhttps://idesr.org Page 5/7International Database of Education Systematic ReviewsResearch question 3: Evaluation of learning-oriented language assessment • Evaluation mechanism/tool • Nature of evaluation (perceptual, observational, effectiveness) • Teacher's perception/experience • Learner's perception/experience • Effectiveness/outcomeRisk of bias/trustworthiness of individual studiesThere are different measures of methodological qualities that are appropriate for different studiesand publication types. Risk of bias and trustworthiness are not likely to be all encompassing. Forexample, assessing methodological quality is not applicable for commentaries or editorials. However,for primary studies and research syntheses, their methodological rigour will be appraised usingexisting appraisal tools/checklists designed specifically for language education research (see"Selection Process"). Data SynthesisA narrative synthesis of the data will be conducted (Petticrew & Roberts, 2006). A narrative synthesisis deemed suitable because of the inclusion of studies from different disciplines and researchparadigms (Schwarz et al., 2019). Specific steps include a preliminary thematic analysis (Nowell etal., 2017) followed by structural consolidation of themes through focused and axial coding (Charmaz,2006). Results will also be presented in tabular form and main findings will be summarised into aliterature-informed framework of learning-oriented language assessment.  Charmaz, K. (2006). Constructing grounded theory. Sage.  Nowell, L. S., Norris, J. M., White, D. E., & Moules, N. J. (2017). Thematic analysis: Striving to meet the trustworthiness criteria. International Journal of Qualitative Methods, 1 6(1), 1-13.https://doi.org/10.1177/1609406917733847 Petticrew, M., & Roberts, H. (2008). Systematic reviews in the social sciences: A practical guide. Blackwell. Schwarz, C. M., Hoffmann, M., Schwarz, P., Kamolz, L., Brunner, G., & Sendlhofer, G. (2019). A systematic literature review and narrative synthesis on the risks of medical discharge letters forpatients' safety. BMC Health Services Research, 19, 158. Meta-biasesConfidence in cumulative evidenceSee "Selection Process".Sources of FundingThis scoping review is funded by British Council through the Assessment Research Award, awardedto the first author.Role of FundersBritish Council provides financial support to the three phases of a study focusing onlearning-oriented language assessment in formal and non-formal EFL contexts: this scoping review, apilot study, and the main study. https://idesr.org Page 6/7International Database of Education Systematic ReviewsAnticipated or actual start date:2022-09-01Anticipated completion date: 2022-12-31Other language resourcesCurrent StatusOngoingDetails of Published ReviewIDESR URLhttps://idesr.org/article/IDESR000040https://idesr.org Page 7/7

A protocol for a scoping review of learning-oriented language assessment in formal and non-formal educational contexts

https://research-repository.st-andrews.ac.uk/bitstream/10023/27910/1/IDESR000040_14_06_2023.pdf

A protocol for a scoping review of learning-oriented language assessment in formal and non-formal educational contexts

Abstract

Similar works

Full text

Available Versions

St Andrews Research Repository