Exploring relationships between automated and human evaluations of L2 texts

Despite the current potential to use computers to automatically generate a large range of text-based indices, many issues remain unresolved about how to apply these data in established language teaching and assessment contexts. One way to resolve these issues is to explore the degree to which automatically generated indices, which are reflective of key measures of text quality, align with parallel measures derived from locally relevant, human evaluations of texts. This study describes the automated evaluation of 104 English as a second language texts through use of the computational tool Coh-Metrix, which was used to generate indices reflecting text cohesion, lexical characteristics, and syntactic complexity. The same texts were then independently evaluated by two experienced human assessors through use of an analytic scoring rubric. The interrelationships between the computer and human generated evaluations of the texts are presented in this paper with a particular focus on the automatically generated indices that were most strongly linked to the human generated measures. A synthesis of these findings is then used to discuss the role that such automated evaluation may have in the teaching and assessment of second language writing

An exploratory study into automated précis grading

Automated writing evaluation is a popular research field, but the main focus has been on evaluating argumentative essays. In this paper, we consider a different genre, namely précis texts. A précis is a written text that provides a coherent summary of main points of a spoken or written text. We present a corpus of English précis texts which all received a grade assigned by a highly-experienced English language teacher and were subsequently annotated following an exhaustive error typology. With this corpus we trained a machine learning model which relies on a number of linguistic, automatic summarization and AWE features. Our results reveal that this model is able to predict the grade of précis texts with only a moderate error margin

A study on digital-based argumentative writing in English of South Korean university students

In higher education in South Korea, English proficiency has been specifically emphasised by the government (Kang, 2015; Kang, 2018; Kim, 2017; Shim & Park, 2008; Williams, 2015). However, writing skills have had little attention in education settings, including higher education institutions (Kim, 2018; Shin, 2018; Park, 2020; Shin & Hyun, 2020; Yu, 2019), despite a series of educational reforms. Students in South Korean higher education are now facing practical and specific needs for argumentative writing in English (Shim, 2016; Shin, 2018). However, the overall context of English education does not fully reflect their real needs (Kim, 2018; Kwon, 2012; Kwak, 2017; Shim, 2016). South Korean universities require their students to reach a specific level at one of the English proficiency tests (Kim, 2018; Ma, 2018; Shim, 2016), most of which include at least one argumentative writing task. Additionally, the certificate of English proficiency test is widely used as the basic skills reference for their career (Kim, 2018). In the meantime, writing proficiency has increasingly gained its own weight in English language tests (Kim, 2018; Ma, 2018; Shin, 2018), adding to the burden on students to develop their writing proficiency (Kim, 2018; Ma, 2018). Despite students’ need for improvement in English writing proficiency, including English argumentative writing, writing courses given by South Korean higher education institutions are still rare (Kim, 2018; Ma, 2018; Shin, 2018; Yu, 2019) and often allow little room for reviewing tasks (Kim, 2018; Ma, 2018; Shim, 2018), even though they commonly use a process-based approach. Furthermore, in immediate response to their needs, higher education institutions in both the public and private sectors have maintained narrow academic attention, focusing on test specific writing skills (Kim, 2018; Shin, 2018). All these situations have resulted in a lack of educational opportunities for students to receive theoretically and systematically well designed instruction in developing their argumentation skills (Shin, 2018). For South Korean students learning English as a foreign language (EFL), argumentative writing in English includes acquiring an understanding of and the skills for both critical thinking and English-specific conventions for the target genre of writing (Ahn & Park, 2019; Choi, 2008; Shim, 2016). To promote a fast and concrete understanding of argumentation in English, representative organisational structures are often used in instructional practices. While many of the courses for English argumentative writing in South Korea are limited to the delivery of instructions, or creating a rough claim-evidence link in a paragraph, this simple formula-based approach may have a limited influence on the level of argumentation that university students in South Korea are able to develop (Choi, 2008). To enhance students’ in-depth knowledge of and skills for making arguments in English, a systematic and effective instructional model is necessary, targeting argumentation development and investigated by rigorous research. However, with a traditionally narrow focus on writing in English education, studies on English writing itself, including argumentative writing, have been limited, despite the importance of this area. As a way of introducing systematically presented models into instruction in English argumentative writing, the Toulmin model can be an effective option. It suggests a detailed, sequenced, intensively explained process for the logical framework for writing in English. In this sense, it is necessary to explore how to modify and apply Toulmin’s components into the courses for English argumentative writing in South Korean higher education. In addition to the practical applicability of the Toulmin model, it is necessary to consider the common context in which writing courses in South Korean higher education institutions provide some phases for drafting and revision/editing, which are broadly anchored in the process-based writing approach. Considering the practical challenge caused by a lack of time for drafting in writing courses (Kim, 2018; Ma, 2018; Shim, 2018), online based classes can be a better option, enabling more flexibility in time and space. Even before the Covid-19 outbreak, diverse synchronous and asynchronous digital writing environments have been utilised in the field of higher education in South Korea to enhance students’ writing performance and also increase the connectivity between learners and teachers. However, the digital environments for English writing in South Korea are still based on a lack of rigour in terms of research evidence, which signals the need for more research into how best to develop digital writing platforms and incorporate necessary support for users. With the two main areas of English argumentative writing and digital learning environments for writing combined, this study explores the effectiveness of a digital-based argumentative writing course in South Korea, as well as the pedagogical implications. To investigate the effects of digital course development for instruction in English argumentative writing for university students in South Korea and derive insights in digital course design for English argumentative writing for university students in South Korea, this study used a sequential mixed-methods design: quantitative phase followed by qualitative phase for collection and analysis of data sets. The English argumentative writing course in this study applies the Toulmin model (1958; 2003) as a specific teaching strategy, with a cycle of drafting and exchanging feedback using the process-based writing approach. To provide the online group with a digital-based collaborative writing1 environment for feedback exchanges, the writing platform, Scholar, was used. In this study, 43 undergraduate students in South Korea participated in a writing course for one semester, 22 participants in a control group (offline course) and 21 participants in an intervention group (online course). They participated in pre- and post-writing tests, two sessions of interviews, and narrative writing for reflection. Also, ten university teaching staff and e-developers took part in one individual interview session each, to provide professional views on the online instructional design that is implemented in the English argumentative writing of this study. In terms of the effectiveness of the online writing course for developing argumentation skills in English, the findings from the quantitative analysis show both online and offline courses had a positive impact on improvement and retention. Although the statistical results present no indication that the online class had higher learning gains than the offline group by any significant difference, this result is supported by the findings from the qualitative analysis, which indicates that the online group performed better in terms of the quality and the quantity of peer feedback. In addition, the findings from the qualitative analysis suggest that the writing course in this study helped students to develop their knowledge and sensitivity in argumentation in English, and the online course facilitated enhanced engagement in feedback tasks. Moreover, despite recognising the value of face-to-face interaction for English argumentative writing, the qualitative findings suggest that the anonymity and convenience of the online writing course in this study encouraged participation in feedback. Finally, the findings from teaching staff and e-developer interviews reveal generally positive perceptions of and evaluations of the usefulness and applicability of the Toulmin model for English argumentation development, and the collaborative writing environment of Scholar. ( 1. In this study, the term, ‘collaborative writing,’ means individual student’s essay writing supported by external feedback, including peer and teacher feedback, not co-authorship in writing one shared essay together.

Developing, Validating, and Incorporating a Rubric for Assessing the Construct of Integration into an EAP Program

Due to their authenticity as an academic writing task, integrated writing tests are widely used for evaluating English for Academic Purposes (EAP) students’ writing ability. However, apart from validation studies focusing on a common set of standardized test rubrics, little research has explored the construct of integration or how to assess it effectively in second language (L2) writing classrooms. Thus, this dissertation designed and validated an analytic rubric for assessing the construct of integration in L2 writing while also examining students’ conceptualizations of integrated writing assessment in an EAP writing course. Study 1 investigated which sub-constructs EAP instructors orient to when assessing L2 writers' integrated essays. Triangulation of data sources from instructor ratings, stimulated recall interviews, and textual analysis of the integrated essays informed the development of an analytic rubric with four categories (i.e., content, organization, source use and language use) for assessing classroom-based integrated writing tasks. Study 2 focused on the validation of the rubric by employing a mixed-methods design, which involved a many-facet Rasch measurement analysis, semi-structured interviews with EAP instructors, and linguistic analysis of student essays for fluency, syntactic and lexical complexity, cohesion, and lexical diversity measures. Results from the Rasch model and textual analysis suggest the rubric is of good quality in terms of assessing one single construct and differentiating the students’ task performance across four levels. The instructor comments during the follow-up interviews contributed to the reformulation of descriptors. Study 3 adopted a case study methodology and investigated L2 learners’ conceptualizations of integrated writing assessment and their use of the analytic rubric for self-assessment in an EAP writing course. Data sources included integrated writing samples that were evaluated by the students and their instructor, a writing self-efficacy questionnaire, individual retrospective interviews, and course materials. Qualitative analysis revealed themes related to three aspects of classroom-based integrated writing assessment: task requirements, task conditions, and instructor feedback. There was an overlap between students’ self-assessment and instructor evaluation of their integrated essays, suggesting that students could use the rubric effectively

The Relationship Between Vocabulary Depth and Argumentative Essay Writing Proficiency Among IELTS Test Takers in Pakistan

The main object of this correlational study was to determine the relationship between vocabulary depth and argumentative essay writing proficiency among Pakistani IELTS test takers. Students with greater vocabulary knowledge and discourse expertise would produce texts of higher quality as compared to students with less of these skills. For this purpose, it manipulated three instruments for collecting the required data: a prompt for writing an argumentative essay, an IELTS band descriptor for scoring writing tasks, and the Word Association Test (WAT) for assessing vocabulary depth knowledge. It involved 129 Pakistani IELTS test-takers as its population. The sample comprised 42 females and 87 male participants. The data was analysed using SPSS version 23. Pearson\u27s product moment analysis was used to determine the relationship between both continuous variables (vocabulary depth and argumentative essay writing), and the results revealed a negative or weak relationship between vocabulary depth and IELTS essay band scores (r = -.032, p =.721). The current findings concluded that there was no statistically significant correlation between both variables. Furthermore, this research study would have certain suggestions for teachers, test designers, and material originators on how to incorporate the dimension of word associations into the creation of word knowledge. It would have some pedagogical implications and instructions for increasing linguistic aptitude levels, which would be useful in academic institutions, language assessment, and language teaching and learning centres such as IELTS training centers. &nbsp

Using Analytic Rubrics to Support Second Language Writing Development in Online Tasks

Ph.D

A Correlational Study of 5th Students\u27 Handwriting Legibility and Scores on Writing Samples in a Northwest Georgia School

This study explored the relationship between legibility in handwriting scores and compositional scores of students in grade five in one Northwest Georgia school. The ability to recall and write the letters automatically may impact the composing skills of students engaged in the writing process. Handwriting, often considered a motor skill in young children, may have a greater impact on literacy learning than is often considered. The strong connection to literacy learning along with the importance as a skill in communications both contribute to the importance of this study. Data was collected from one elementary school in Northwest Georgia. The school was chosen based on location and the school’s use of a writing workshop model. As one school was chosen, all students are a part of the sampling for this study. A writing rubric for each piece of writing was scored and then compared to students’ handwriting scores to determine if there is a statistically significant correlation. Sufficient evidence during this study to reject both null hypothesis was found. The results of this correlational study can add to the body of research investigating the amount of instructional time spent on handwriting

Development and validation of virtual interactive tasks for an aviation English assessment

In response to growing concerns over aviation safety stemming from the limited command of aviation English by non-native English speaking practitioners, this study aimed to demonstrate the development process of aviation English test tasks in a virtual environment and investigate the validity for a task-based aviation English performance assessment in the context of Korean Army Aviation. In the current dissertation study, the development and validation of the VITAEA test were based on four inferences – domain description, evaluation, generalization, and explanation – and underlying assumptions in an interpretive argument that developed with reference to argument-based validity, evidence-centered design, target language use situation analysis for test development in language for specific purposes, and task-based language assessment. Adopting a mixed method with a triangulation design, qualitative and quantitative evidence was collected to provide valid support for the inferences and to strengthen the validity argument. Based on a task-based needs analysis with 81 military air traffic controllers on the required aviation English knowledge, skills, processes, target tasks, and task procedures in the TLU situations, virtual interactive aviation English tasks were developed in Second Life. A total of 20 controllers completed the prototype virtual interactive tasks for aviation English assessment, and their output was then rated by two rater groups, one engaging in task-centered rating and one accomplishing language-centered rating. Data included 20 task-based performance assessment sample audio files; 19 follow-up test taker interviews and online survey questionnaires; three language-centered raters\u27 post-rating questionnaire responses; two task-centered raters\u27 post-rating interview transcripts; military aviation English training manuals and references; and coded transcripts of 12 test takers\u27 stimulated recall and their actual task performance. The validity evidence collected in the various phases of test development and validation serves as backing for the four inferences in the interpretive argument as well as provides invaluable resources for the revision of the prototype virtual interactive tasks for aviation English assessment. Furthermore, empirical processes for prototype test development and partial validation based on the theoretical guidance presented in this dissertation study can be seen as among the first to be constructed utilizing the three theoretical frameworks – argument-based approach, evidence-centered design, and task-based language assessment. In addition, this dissertation study can shed light on the steps required in application of an argument-based approach for task-based second language assessment. Lastly, this study provides additional grounds for the potential use of an immersive interface and simulated target language use situations in virtual environments to provide test takers with more authentic opportunities to perform the target tasks

Evaluation of ChatGPT Feedback on ELL Writers' Coherence and Cohesion

Since its launch in November 2022, ChatGPT has had a transformative effect on education where students are using it to help with homework assignments and teachers are actively employing it in their teaching practices. This includes using ChatGPT as a tool for writing teachers to grade and generate feedback on students' essays. In this study, we evaluated the quality of the feedback generated by ChatGPT regarding the coherence and cohesion of the essays written by English Language Learners (ELLs) students. We selected 50 argumentative essays and generated feedback on coherence and cohesion using the ELLIPSE rubric. During the feedback evaluation, we used a two-step approach: first, each sentence in the feedback was classified into subtypes based on its function (e.g., positive reinforcement, problem statement). Next, we evaluated its accuracy and usability according to these types. Both the analysis of feedback types and the evaluation of accuracy and usability revealed that most feedback sentences were highly abstract and generic, failing to provide concrete suggestions for improvement. The accuracy in detecting major problems, such as repetitive ideas and the inaccurate use of cohesive devices, depended on superficial linguistic features and was often incorrect. In conclusion, ChatGPT, without specific training for the feedback generation task, does not offer effective feedback on ELL students' coherence and cohesion.Comment: 24 pages, 1 figure

Does Lexical Frequency affect rater judgement of essays? An experimental design using quantitative and qualitative data

Many correlational studies show a positive relation between written assessments of language and use of more diverse vocabulary (Lexical Diversity) and more infrequent words (Lexical Frequency). However, there have been no experimental studies that have isolated the effects of Lexical Frequency from Lexical Diversity. In the present study, 14 raters judged two versions of the same essay that differed only in Lexical Frequency. A Paired T-test showed no difference in mean scores between essays (t(13) = .396, p = .70) when the Lexical Frequency of 23.5% of Content Words were changed in a 347 word essay. Comments explaining scores given to essays showed that features other than vocabulary had a far greater influence on rater judgement. It is possible that the Lexical Frequency manipulations were not great enough to affect rater judgement, whether subliminal or conscious. Implications of these results for standardized language proficiency tests and future research in vocabulary are discussed