A robust methodology for automated essay grading

None of the available automated essay grading systems can be used to grade essays according to the National Assessment Program – Literacy and Numeracy (NAPLAN) analytic scoring rubric used in Australia. This thesis is a humble effort to address this limitation. The objective of this thesis is to develop a robust methodology for automatically grading essays based on the NAPLAN rubric by using heuristics and rules based on English language and neural network modelling

Guidance on the principles of language accessibility in National Curriculum Assessments : research background

This review accompanies the document, which describes the principles which should guide the development of clear assessment questions. The purpose of the review is to present and discuss in detail the research underpinning these principles. It begins from the standpoint that National Curriculum assessments, indeed any assessments, should be: - appropriate to the age of the pupils - an effective measure of their abilities, skills and concept development - fair to all irrespective of gender, language, religion, ethnic or social origin or disability. (Ofqual, 2011) The Regulatory Framework for National Assessments: National Curriculum and Early Years Foundation Stage (Ofqual, 2011) sets out a number of common criteria which apply to all aspects of the development and implementation of National Assessments. One of these criteria refers to the need for assessment procedures to minimise bias: “The assessment should minimise bias, differentiating only on the basis of each learner’s ability to meet National Curriculum requirements” (Section 5.39, page 16). The Framework goes on to argue that: “Minimising bias is about ensuring that an assessment does not produce unreasonably adverse outcomes for particular groups of learners” (Annex 1, page 29). This criterion reinforces the guiding principle that any form of assessment should provide information about the knowledge and understanding of relevant content material. That is to say that the means through which this knowledge and understanding is examined, the design of the assessment and the language used should as far as possible be transparent, and should not influence adversely the performance of those being assessed. There is clearly a large number of ways in which any given assessment task can be presented and in which questions can be asked. Some of these ways will make the task more accessible – that is, easier to complete successfully – and some will get in the way of successful completion. Section 26 of the Fair Access by Design (Ofqual, 2010) document lists a number of guiding principles for improving the accessibility of assessment questions, although the research basis for these principles is not made completely clear in that document. The aim of the current review is to examine the research background more closely in order to provide a more substantial basis for a renewed set of principles to underpin the concept of language accessibility. In the review, each section will be prefaced by a statement of the principles outlined in Guidance on the Principles of Language Accessibility in National Curriculum Assessments and then the research evidence underpinning these principles will be reviewed

The role of feedback in the processes and outcomes of academic writing in english as a foreign language at intermediate and advanced levels

Providing feedback on students’ texts is one of the essential components of teaching second language writing. However, whether and to what extent students benefit from feedback has been an issue of considerable debate in the literature. While many researchers have stressed its importance, others expressed doubts about its effectiveness. Regardless of these continuing and well-established debates, instructors consider feedback as a worthwhile pedagogical practice for second language learning. Based on this premise, I conducted three experimental studies to investigate the role of written feedback in Myanmar and Hungarian tertiary EFL classrooms. Additionally, I studied syntactic features and language-related error patterns in Hungarian and Myanmar students’ writing. This attempt was made to understand how students with different writing proficiency acted upon teacher and automated feedback. The first study examined the efficacy of feedback on Myanmar students’ writing over a 13-week semester and how automated feedback provided by Grammarly could be integrated into writing instruction as an assistance tool for writing teachers. Results from pre-and post-tests demonstrated that students’ writing performance improved along the lines of four assessment criteria: task achievement, coherence and cohesion, grammatical range and accuracy, and lexical range and accuracy. Further results from a written feedback analysis revealed that the free version of Grammarly provided feedback on lower-level writing issues such as articles and prepositions, whereas teacher feedback covered both lower-and higher-level writing concerns. These findings suggested a potential for integrating automated feedback into writing instruction. As limited attention was given to how feedback influences other aspects of writing development beyond accuracy, the second study examined how feedback influences the syntactic complexity of Myanmar students’ essays. Results from paired sample t-tests revealed no significant differences in the syntactic complexity of students’ writing when the comparison was made between initial and revised texts and between pre-and post-tests. These findings suggested that feedback on students’ writing does not lead them to write less structurally complex texts despite not resulting in syntactic complexity gains. The syntactic complexity of students’ revised texts varied among high-, mid-, and low-achieving students. These variations could be attributed to proficiency levels, writing prompts, genre differences, and feedback sources. The rationale for conducting the third study was based on the theoretical orientation that differential success in learners’ gaining from feedback largely depended on their engagement with the feedback rather than the feedback itself. Along these lines of research, I examined Hungarian students’ behavioural engagement (i.e., students’ uptake or revisions prompted by written feedback) with teacher and automated feedback in an EFL writing course. In addition to the engagement with form-focused feedback examined in the first study, I considered meaning-focused feedback, as feedback in a writing course typically covers both linguistic and rhetorical aspects of writing. The results showed differences in feedback focus (the teacher provided form-and meaning-focused feedback) with unexpected outcomes: students’ uptake of feedback resulted in moderate to low levels of engagement with feedback. Participants incorporated more form-focused feedback than meaning-focused feedback into their revisions. These findings contribute to our understanding of students’ engagement with writing tasks, levels of trust, and the possible impact of students’ language proficiency on their engagement with feedback. Following the results that Myanmar and Hungarian students responded to feedback on their writing differently, I designed a follow-up study to compare syntactic features of their writing as indices of their English writing proficiency. In addition, I examined language-related errors in their texts to capture the differences in the error patterns in the two groups. Results from paired sample t-tests showed that most syntactic complexity indices distinguished the essays produced by the two groups: length of production units, sentence complexity, and subordination indices. Similarly, statistically significant differences were found in language-related error patterns in their texts: errors were more prevalent in Myanmar students’ essays. The implications for research and pedagogical practices in EFL writing classes are discussed with reference to the rationale for each study

The Effect of Constant Time Delay and Sentence Frames on Correct Word Selection for Sentences Constructed Using Technology-Aided Instruction During a Story-Based Lesson

The purpose of this study was to investigate the effect of an intervention package that included constant time delay (CTD) and sentence frames on correct word selection for sentences constructed using technology-aided instruction (TAI) during and following a story-based lesson (SBL) for participants ages 6-8 who have intellectual disability, developmental delays, and autism (IDD). A multiple probe across participants design was used to evaluate the efficacy of the intervention. Probes were conducted during baseline, intervention, generalization, and maintenance phases to determine the effectiveness of the intervention on correct word selection for sentence construction. Results indicate a functional relation between the intervention package on correct word selection for sentence construction during a shared story. Additionally, the Percent of Nonoverlapping Data (PND) indicate overall strong effects. Finally, students were able to demonstrate both generalization and maintenance of skills

Testing the use of grammar: Beyond grammatical accuracy

Udostępnienie publikacji Wydawnictwa Uniwersytetu Łódzkiego finansowane w ramach projektu „Doskonałość naukowa kluczem do doskonałości kształcenia”. Projekt realizowany jest ze środków Europejskiego Funduszu Społecznego w ramach Programu Operacyjnego Wiedza Edukacja Rozwój; nr umowy: POWER.03.05.00-00-Z092/17-00

Linguistic and Cognitive Measures in Arabic-Speaking English Language Learners (ELLs) and monolingual children with and without Developmental Language Disorder (DLD)

Understanding the current level of language knowledge in English Language Learners (ELLs) can present a challenge. The standardized language tests that are commonly used to assess language tap prior knowledge and experience. ELLs may score poorly on such ‘knowledge-based’ measures because of the low levels of exposure to each of their languages. Considerable overlap has been found on several knowledge-based measures (Paradis, 2010) between ELLs and monolingual children with an unexpected delay in language development known as Developmental Language Disorder (DLD). Measures of cognitive processing, on the other hand, are less dependent on ELLs’ linguistic knowledge because they employ nonlinguistic or novel stimuli to tap skills considered to underlie language learning. It has been suggested that processing-dependent tasks such as measures of verbal short-term memory may differentiate ELLs from children with DLD (Kohnert, Windsor, & Yim, 2006; Paradis, Schneider, & Duncan, 2013). This thesis presents three studies that investigated the performance of Arabic-speaking ELLs and monolingual children with and without DLD on linguistic and cognitive measures. Study 1 provided a description of the performance of monolingual Arabic-speaking children on a battery of Arabic language tests. The results of study 1 revealed that the majority of language measures were sensitive to developmental change in younger children between the ages of 6 and 7. Study 2 demonstrated lower standardized scores by ELLs on the Arabic and English knowledge-based language tasks. However, ELLs scored above or at age-level expectations on the cognitive measures, with the exception of an Arabic-nonword repetition task. Study 3 found a significant overlap between ELLs and monolingual Arabic-speaking children with DLD on first language (L1) knowledge-based measures. With the exception of the Arabic nonword repetition task, verbal short-term and working memory tasks distinguished ELLs from children with underlying language impairment. The results indicated that there is a need to develop language assessment measures that evaluate a broad range of language abilities for Arabic-speaking children. The findings also suggested that unlike knowledge-based measures, cognitive measures may be valid assessment tools that minimize the role of linguistic knowledge and experiences and help distinguish between ELLs and children with DLD