Rater Cognition in L2 Speaking Assessment: A Review of the Literature

This literature review attempts to survey representative studies within the context of L2 speaking assessment that have contributed to the conceptualization of rater cognition. Two types of studies are looked at: 1) studies that examine how raters differ (and sometimes agree) in their cognitive processes and rating behaviors, in terms of their focus and feature attention, their approaches to scoring, and their treatment of the scoring criteria and non-criteria relevant aspects and features of the speaking performance; 2) studies that explore why raters differ, through the analysis of the interactions between several rater background factors (i.e., rater language background, rater experience and rater training) and their rating behaviors and decision-making processes. The two types of studies have improved our understanding of the nature and the causes of rater variability in their perception and evaluation of L2 speech. However, very few of those studies has drawn on existing theories of human information processing and research on strategy use, which can explain on a cognitive-processing (Purpura, 2014) level what goes on in raters’ mind during assessment. It is argued as a final conclusion that only based on established frameworks of human information processing and research on (meta)cognitive strategy use can rater cognition be explored with more depth and breadth

The effects of automated essay scoring as a high school classroom intervention

This quasi-experimental, mixed methods study investigated whether students writing development and proficiency, in combination with teacher-led instruction, are significantly affected by the use of an automated essay scoring (AES) system. The ninth grade standard and honors English students were divided into control and treatment groups at a large, urban high school. Student writing was examined for any changes in proficiency, measured by human- and AES-scored holistic measures. A developmental writing index was used to analyze the rate of change in pre- and post-essays. The AES system was further researched by comparing the treatment and control groups\u27 trait score categories. Finally, treatment students were interviewed and surveyed to identify their degree of satisfaction with the AES system; Automated essay scoring systems have moved from their original purpose of rapidly and reliably scoring high stakes testing into the classroom as an instructional tool providing holistic and trait scoring. One area of potential AES usefulness is to provide students with more writing opportunities that include feedback. While supporting research findings that student writing improves if more writing opportunities with feedback are provided, this also supports the iterative process of writing and revision; To support teachers\u27 optimum classroom technology integration of an AES system to supplement teacher-led instruction, an access ratio of one Internet-connected computer for each student, (i.e., 1:1) needs to be provided. System-provided or teacher-provided writing prompts (i.e., topics) can be selected to provide students with AES simulations of the summative score of high stakes testing, in concert with formative trait scoring, which gives specific recommendations to improve writing; No gender difference was shown for the treatment participants from the AES-scored measures. The human-scored writing proficiency and development measures were inconclusive for gender and class levels due to the small sample size. By class levels, treatment honors students performed significantly better on the AES-scored proficiency measure, but the results were not supported by the human-scored measure. The other AES-scored measures analyzed by class levels, the development and trait category measures, did not show significance. However, the treatment participants expressed a high degree of satisfaction with the use of the AES system

Rise of the machines? The evolving role of Artificial Intelligence (AI) technologies in high stakes assessment

Our world has been transformed by technologies incorporating artificial intelligence (AI) within mass communication, employment, entertainment and many other aspects of our daily lives. However, within the domain of education, it seems that our ways of working and, particularly, assessing have hardly changed at all. We continue to prize examinations and summative testing as the most reliable way to assess educational achievements, and we continue to rely on paper-based test delivery as our modus operandi. Inertia, tradition and aversion to perceived risk have resulted in a lack of innovation (James, 2006), particularly so in the area of high-stakes assessment. The summer of 2020 brought this deficit into very sharp focus with the A-level debacle in England, where grades were awarded, challenged, rescinded and reset. These events are potentially catastrophic in terms of how we trust national examinations, and the problems arise from using just one way to define academic success and one way to operationalize that approach to assessment. While sophisticated digital learning platforms, multimedia technologies and wireless communication are transforming what, when and how learning can take place, transformation in national and international assessment thinking and practice trails behind. In this article, we present some of the current research and advances in AI and how these can be applied to the context of high-stakes assessment. Our discussion focuses not on the question of whether we should be using technologies, but on how we can use them effectively to better support practice. An example from one testing agency in England using a globally popular test of English that assesses oral, aural, reading and written skills is described to explain and propose just how well new technologies can augment assessment theory and practice

The Road Ahead for State Assessments

The adoption of the Common Core State Standards offers an opportunity to make significant improvements to the large-scale statewide student assessments that exist today, and the two US DOE-funded assessment consortia -- the Partnership for the Assessment of Readiness for College and Careers (PARCC) and the SMARTER Balanced Assessment Consortium (SBAC) -- are making big strides forward. But to take full advantage of this opportunity the states must focus squarely on making assessments both fair and accurate.A new report commissioned by the Rennie Center for Education Research & Policy and Policy Analysis for California Education (PACE), The Road Ahead for State Assessments, offers a blueprint for strengthening assessment policy, pointing out how new technologies are opening up new possibilities for fairer, more accurate evaluations of what students know and are able to do. Not all of the promises can yet be delivered, but the report provides a clear set of assessment-policy recommendations. The Road Ahead for State Assessments includes three papers on assessment policy.The first, by Mark Reckase of Michigan State University, provides an overview of computer adaptive assessment. Computer adaptive assessment is an established technology that offers detailed information on where students are on a learning continuum rather than a summary judgment about whether or not they have reached an arbitrary standard of "proficiency" or "readiness." Computer adaptivity will support the fair and accurate assessment of English learners (ELs) and lead to a serious engagement with the multiple dimensions of "readiness" for college and careers.The second and third papers give specific attention to two areas in which we know that current assessments are inadequate: assessments in science and assessments for English learners.In science, paper-and-pencil, multiple choice tests provide only weak and superficial information about students' knowledge and skills -- most specifically about their abilities to think scientifically and actually do science. In their paper, Chris Dede and Jody Clarke-Midura of Harvard University illustrate the potential for richer, more authentic assessments of students' scientific understanding with a case study of a virtual performance assessment now under development at Harvard. With regard to English learners, administering tests in English to students who are learning the language, or to speakers of non-standard dialects, inevitably confounds students' content knowledge with their fluency in Standard English, to the detriment of many students. In his paper, Robert Linquanti of WestEd reviews key problems in the assessment of ELs, and identifies the essential features of an assessment system equipped to provide fair and accurate measures of their academic performance.The report's contributors offer deeply informed recommendations for assessment policy, but three are especially urgent.Build a system that ensures continued development and increased reliance on computer adaptive testing. Computer adaptive assessment provides the essential foundation for a system that can produce fair and accurate measurement of English learners' knowledge and of all students' knowledge and skills in science and other subjects. Developing computer adaptive assessments is a necessary intermediate step toward a system that makes assessment more authentic by tightly linking its tasks and instructional activities and ultimately embedding assessment in instruction. It is vital for both consortia to keep these goals in mind, even in light of current technological and resource constraints.Integrate the development of new assessments with assessments of English language proficiency (ELP). The next generation of ELP assessments should take into consideration an English learners' specific level of proficiency in English. They will need to be based on ELP standards that sufficiently specify the target academic language competencies that English learners need to progress in and gain mastery of the Common Core Standards. One of the report's authors, Robert Linquanti, states: "Acknowledging and overcoming the challenges involved in fairly and accurately assessing ELs is integral and not peripheral to the task of developing an assessment system that serves all students well. Treating the assessment of ELs as a separate problem -- or, worse yet, as one that can be left for later -- calls into question the basic legitimacy of assessment systems that drive high-stakes decisions about students, teachers, and schools." Include virtual performance assessments as part of comprehensive state assessment systems. Virtual performance assessments have considerable promise for measuring students' inquiry and problem-solving skills in science and in other subject areas, because authentic assessment can be closely tied to or even embedded in instruction. The simulation of authentic practices in settings similar to the real world opens the way to assessment of students' deeper learning and their mastery of 21st century skills across the curriculum. We are just setting out on the road toward assessments that ensure fair and accurate measurement of performance for all students, and support for sustained improvements in teaching and learning. Developing assessments that realize these goals will take time, resources and long-term policy commitment. PARCC and SBAC are taking the essential first steps down a long road, and new technologies have begun to illuminate what's possible. This report seeks to keep policymakers' attention focused on the road ahead, to ensure that the choices they make now move us further toward the goal of college and career success for all students. This publication was released at an event on May 16, 2011

Assessment of EFL speaking skills in Qatari public secondary Schools: teachers' practices and challenges

This thesis aims to conduct a quantitative investigation into the practices and challenges of EFL teachers in assessing their students’ speaking skills. To collect data for this study, all EFL teachers currently working for the Ministry of Education and Higher Education in Qatar were invited to participate in an online survey using Google Forms Software. A total of 120 teachers took part in the data collection process by completing the questionnaire. Using SPSS 23 Software, the data was analyzed under five sets of assessment practices and three categories of challenges. Descriptive statistics revealed that EFL teachers were committed to providing enough time for the assessment of students’ EFL speaking skills. In addition, results proved that teachers were careful to differentiate speaking assessment tasks, use a rating scale in scoring students’ performance and provide students with feedback. However, teachers’ challenges in the assessment of EFL speaking skills were mainly related to practicality issues, the lack of relevant training and the students’ low levels of motivation and English proficiency

Developing an Automated Test of Spoken Japanese

In order to assess spoken skills of learners of Japanese effectively and more efficiently the Institute for DECODE (Institute for Digital Enhancement of Cognitive Development) at Waseda University is collaborating with Ordinate Corporation to develop and validate an automated test of spoken Japanese, SJT (Spoken Japanese Test). The SJT is intended to measure a test-taker’s facility in spoken Japanese, that is listening and speaking skills in daily conversation, in a quick, accurate and reliable manner. In this paper, we discuss the purposes for developing the SJT, the mechanism of a fully automated test, and the test development processes, including item development and implementation