Using Natural Language Processing to Increase Modularity and Interpretability of Automated Essay Evaluation and Student Feedback

For English teachers and students who are dissatisfied with the one-size-fits-all approach of current Automated Essay Scoring (AES) systems, this research uses Natural Language Processing (NLP) techniques that provide a focus on configurability and interpretability. Unlike traditional AES models which are designed to provide an overall score based on pre-trained criteria, this tool allows teachers to tailor feedback based upon specific focus areas. The tool implements a user-interface that serves as a customizable rubric. Students’ essays are inputted into the tool either by the student or by the teacher via the application’s user-interface. Based on the rubric settings, the tool evaluates the essay and provides instant feedback. In addition to rubric-based feedback, the tool also implements a Multi-Armed Bandit recommender engine to suggest educational resources to the student that align with the rubric. Thus, reducing the amount of time teachers spend grading essay drafts and re-teaching. The tool developed and deployed as part of this research reduces the burden on teachers and provides instant, customizable feedback to students. Our minimum estimation for time savings to students and teachers is 117 hours per semester. The effectiveness of the feedback criteria for predicting if an essay was proficient or needs improvement was measured using recall. The recall for the model built for the persuasive essays was 0.96 and 0.86 for the source dependent essay model

A Corpus-Based Analysis of Cohesion in L2 Writing by Undergraduates in Ecuador

In finding out the nature of cohesion in L2 writing, the present study set out to address three research questions: (1) What types of cohesion relations occur in L2 writing at the sentence, paragraph, and whole-text levels? (2) What is the relationship between lexico-grammatical cohesion features and teachers’ judgements of writing quality? (3) Do expectations of cohesion suggested by the CEFR match what is found in student writing? To answer those questions, a corpus of 240 essays and 240 emails from college- level students learning English as a foreign language in Ecuador enabled the analysis of cohesion. Each text included the scores, or teachers’ judgements of writing quality aligned to the upper-intermediate level (or B2) as proposed by the Common European Framework of Reference for learning, teaching, and assessing English as a foreign language. Lexical and grammatical items used by L2 students to build relationships of meaning in sentences, paragraphs, and the entire text were considered to analyse cohesion in L2 writing. Utilising Natural Language Processing tools (e.g., TAACO, TextInspector, NVivo), the analysis focused on determining which cohesion features (e.g., word repetition/overlap, semantical similarity, connective words) predicted the teachers’ judgements of writing quality in the collected essays and emails. The findings indicate that L2 writing is characterised by word overlap and synonyms occurring at the paragraph level and, to a lesser degree, cohesion between sentences and the entire text (e.g., connective words). Whilst these cohesion features positively and negatively predicted the teachers’ scores, a cautious interpretation of these findings is required, as many other factors beyond cohesion features must have also influenced the allocation of scores in L2 writing

EDIT: an Educational Design Intelligence Tool for supporting design decisions

Designing for learning is a complex task and considered one of the most fundamental activities of teaching practitioners. A well-balanced teaching system ensures that all aspects of teaching, from the intended learning outcomes, the teaching and learning activities used, and the assessment tasks are all associated and aligned to each other (Biggs, 1996). This guarantees appropriate and therefore effective student engagement. The design and promotion of constructively aligned teaching practices has been supported to some degree by the development of software tools that attempt to support teaching practitioners in the design process and assist them in the development of more informed design decisions. Despite the potential of the existing tools, these tools have several limitations in respect of the support and guidance provided and cannot be adapted according to how the design pattern works in practice. Therefore; there is a real need to incorporate an intelligent metric system that enables intelligent design decisions to be made not only theoretically according to pedagogical theories but also practically based on good design practices according to high levels of satisfaction scores. To overcome the limitations of existing design tools, this research explores machine learning techniques; in particular artificial neural networks as an innovative approach for building an Educational Intelligence Design Tool EDIT that supports teaching practitioners to measure, align, and edit their teaching designs based on good design practices and on the pedagogic theory of constructive alignment. Student satisfaction scores are utilized as indicators of good design practice to identify meaningful alignment ranges for the main components of Tepper's metric (2006). It is suggested that modules designed within those ranges will be well-formed and constructively aligned and potentially yield higher student satisfaction. On this basis, the research had developed a substantial module design database with 519 design patterns spanning 476 modules from the STEM discipline. This is considered the first substantial database compared to the state-of-the-art Learning Design Support Environment (LDSE)(Laurillard, 2011), which includes 122 design patterns available. In order to have a neural-based framework for EDIT, a neural auto-encoder was incorporated to act as an auto-associative memory that learns on the basis of exposure to sets of 'good' design patterns. 519 generated design patterns were coded as input criteria and introduced to the designed neural network with feed-forward multilayer perceptron architecture using the IV hyperbolic tangent function and back-propagation training algorithm for learning the desired task. After successful training (88%), the testing phase was followed by presenting 102 new patterns (associated with low student satisfaction) to the network where higher pattern errors were generated suggesting substantial design changes to input patterns had been generated by the network. The findings of the research are significant in showing the degree of changes for the test patterns (before) and (after) and evaluating the relationships between the core features of module designs and overall student satisfaction. T-test analysis results show statistically significant differences in the test set (before) and (after) in case of the alignment score between learning outcomes and learning objectives (V1) and the alignment score between learning objectives and teaching activities (V2), whereas no statistically significant difference is seen in the alignment score between learning outcomes and assessment tasks (V3). The network gives an average improvement of 0.9, 1.5, and 0.5 in the alignment scores of V1, V2, and V3, respectively. This resulted in increasing the average of satisfaction scores from 3.3 to 3.8. Accordingly, positive correlation with different degrees between student satisfaction and the alignment scores were suggested as a result of applying the network proposal changes. EDIT, with its data‐orientated and adaptive approach to design, reveals orthodox practices whilst revealing some unexpected incongruity between alignment theory and design practice. For example, as expected, increasing the amount of questioning, interaction and group‐based activity effects higher levels of student satisfaction even though misalignment may be present. However, the model is relatively ambivalent towards the alignment of learning outcomes and learning objectives suggesting there is some confusion between practitioners as to how these are related. Also, this confusion appears to persist when defining session learning objectives for different types of teaching, learning and assessment tasks in that the activities themselves appear to be at a higher cognitive level according to Bloom's Taxonomy than the respective learning objectives (resulting in positive misalignment)

Examining the Role of Linguistic Flexibility in the Text Production Process

abstract: A commonly held belief among educators, researchers, and students is that high-quality texts are easier to read than low-quality texts, as they contain more engaging narrative and story-like elements. Interestingly, these assumptions have typically failed to be supported by the writing literature. Research suggests that higher quality writing is typically associated with decreased levels of text narrativity and readability. Although narrative elements may sometimes be associated with high-quality writing, the majority of research suggests that higher quality writing is associated with decreased levels of text narrativity, and measures of readability in general. One potential explanation for this conflicting evidence lies in the situational influence of text elements on writing quality. In other words, it is possible that the frequency of specific linguistic or rhetorical text elements alone is not consistently indicative of essay quality. Rather, these effects may be largely driven by individual differences in students' ability to leverage the benefits of these elements in appropriate contexts. This dissertation presents the hypothesis that writing proficiency is associated with an individual's flexible use of text properties, rather than simply the consistent use of a particular set of properties. Across three experiments, this dissertation relies on a combination of natural language processing and dynamic methodologies to examine the role of linguistic flexibility in the text production process. Overall, the studies included in this dissertation provide important insights into the role of flexibility in writing skill and develop a strong foundation on which to conduct future research and educational interventions.Dissertation/ThesisDoctoral Dissertation Psychology 201

Is a Seat at the Table Enough? Engaging Teachers and Students in Dataset Specification for ML in Education

Despite the promises of ML in education, its adoption in the classroom has surfaced numerous issues regarding fairness, accountability, and transparency, as well as concerns about data privacy and student consent. A root cause of these issues is the lack of understanding of the complex dynamics of education, including teacher-student interactions, collaborative learning, and classroom environment. To overcome these challenges and fully utilize the potential of ML in education, software practitioners need to work closely with educators and students to fully understand the context of the data (the backbone of ML applications) and collaboratively define the ML data specifications. To gain a deeper understanding of such a collaborative process, we conduct ten co-design sessions with ML software practitioners, educators, and students. In the sessions, teachers and students work with ML engineers, UX designers, and legal practitioners to define dataset characteristics for a given ML application. We find that stakeholders contextualize data based on their domain and procedural knowledge, proactively design data requirements to mitigate downstream harms and data reliability concerns, and exhibit role-based collaborative strategies and contribution patterns. Further, we find that beyond a seat at the table, meaningful stakeholder participation in ML requires structured supports: defined processes for continuous iteration and co-evaluation, shared contextual data quality standards, and information scaffolds for both technical and non-technical stakeholders to traverse expertise boundaries

Preparing Students for College and Careers

Preparing Students for College and Careers addresses measurement and research issues related to college and career readiness. Educational reform efforts across the United States have increasingly taken aim at measuring and improving postsecondary readiness. These initiatives include developing new content standards, redesigning assessments and performance levels, legislating new developmental education policy for colleges and universities, and highlighting gaps between graduates’ skills and employers’ needs. In this comprehensive book, scholarship from leading experts on each of these topics is collected for assessment professionals and for education researchers interested in this new area of focus. Cross-disciplinary chapters cover the current state of research, best practices, leading interventions, and a variety of measurement concepts, including construct definitions, assessments, performance levels, score interpretations, and test uses

Looking smart is not the ultimate goal: An examination of a gifted and talented science program

At Metropolitan High School (MHS) a gifted and talented science program (GTSP) operates to meet the educational needs of exceptional students. Academic achievement is dependent on many factors including cognitive ability, goal orientation, selfregulation of learning and self-efficacy. Few studies have attempted to investigate the significance of each of these in special populations particularly in the academic field of science. The literature indicates that educational programs should be subject to evaluation, yet such evaluation is not routinely carried out. In a balanced teaching system, components such as curriculum, teaching methods, assessment procedures and classroom environment are aligned so that they complement each other to create the desired outcomes. The aim of this research was to investigate whether components of the GTSP were aligned to promote a deep approach to learning and the use of self-regulated learning strategies which are important intrapersonal catalysts in Gagné‟s model of giftedness and talents. In the pragmatist paradigm, quantitative and qualitative data forms were utilised to allow methodological triangulation to enhance the rigor of the research process. The research was an exploratory, parallel, nested, mixed model study. Data were integrated at the analysis phase to examine the GTSP, the object of the case study. Within the GTSP best practice education for the gifted was balanced against the requirements of the MHS science curriculum. GTSP students demonstrated high level outcomes in school, state and national measures of science achievement despite the fact that participation in the GTSP did not facilitate a significant increase in deep learning. In order to promote deep learning, self-regulation and the high achievement of GTSP students into the future, it is recommended that the assessment practices within the GTSP are reviewed and aligned with best practice education for the gifted

Preparing Students for College and Careers

Preparing Students for College and Careers addresses measurement and research issues related to college and career readiness. Educational reform efforts across the United States have increasingly taken aim at measuring and improving postsecondary readiness. These initiatives include developing new content standards, redesigning assessments and performance levels, legislating new developmental education policy for colleges and universities, and highlighting gaps between graduates’ skills and employers’ needs. In this comprehensive book, scholarship from leading experts on each of these topics is collected for assessment professionals and for education researchers interested in this new area of focus. Cross-disciplinary chapters cover the current state of research, best practices, leading interventions, and a variety of measurement concepts, including construct definitions, assessments, performance levels, score interpretations, and test uses

Alignment of intended learning outcomes, curriculum and assessment in a middle school science program

This study focused on the intended learning outcomes, curriculum and assessment in the science curriculum offered at a regional independent Middle School in the state of Victoria, Australia. In-school assessment has indicated that the current science curriculum of this Middle School may not develop students\u27 skills in scientific literacy as effectively as intended. One hypothesis to explain this deficit is that there is a misalignment of intended outcomes, curriculum materials and assessment. This study aimed to determine the extent to which the intended curriculum and assessment in this Victorian middle years\u27 science program is aligned to its stated goals and objectives and to design, implement and evaluate a model for assessing the degree of alignment of intended outcomes, curriculum and assessment. Participants in the study were asked to analyse curriculum materials and assessment tasks from two different science courses at the case study school. These curriculum materials and assessments were scored against a series of instruments adapted from curriculum evaluation models used in previous research. The reviewers scored the material to determine the degree of alignment between the intended outcomes, curriculum materials and assessment tasks. The data provided an insight into both the degree of alignment of the curriculum as well as the features of strongly aligned curriculum materials. The effectiveness of the evaluation model was determined by analysis of the scoring data and semi-structured interviews with the participants. The current investigation established that the case study Middle School science program had some degree of alignment, but there were a number of materials and tasks which were not adequately aligned. The features of the curriculum materials and assessment tasks generally matched those identified in the literature, and provided the basis for potential reform to increase the degree of alignment in intended curriculum and assessment in science courses designed to address scientific literacy. The study also demonstrated that the model of curriculum evaluation was effective in establishing the alignment of curriculum materials and assessment with intended goals, particularly when enacted by teachers and administrators within the school context who had been trained. The curriculum analysis can highlight areas of the science curriculum which are not aligned and hence focus curriculum reform efforts