Using keystroke logging to understand writers’ processes on a reading-into-writing test

Background Integrated reading-into-writing tasks are increasingly used in large-scale language proficiency tests. Such tasks are said to possess higher authenticity as they reflect real-life writing conditions better than independent, writing-only tasks. However, to effectively define the reading-into-writing construct, more empirical evidence regarding how writers compose from sources both in real-life and under test conditions is urgently needed. Most previous process studies used think aloud or questionnaire to collect evidence. These methods rely on participants’ perceptions of their processes, as well as their ability to report them. Findings This paper reports on a small-scale experimental study to explore writers’ processes on a reading-into-writing test by employing keystroke logging. Two L2 postgraduates completed an argumentative essay on computer. Their text production processes were captured by a keystroke logging programme. Students were also interviewed to provide additional information. Keystroke logging like most computing tools provides a range of measures. The study examined the students’ reading-into-writing processes by analysing a selection of the keystroke logging measures in conjunction with students’ final texts and interview protocols. Conclusions The results suggest that the nature of the writers’ reading-into-writing processes might have a major influence on the writer’s final performance. Recommendations for future process studies are provided

Human or AI? Using Digital Behavior to Verify Essay Authorship

Large language models (LLMs) such as OpenAI\u27s GPT-4 have transformed natural language processing with their ability to understand context and generate human-like text. This has led to considerable debate, especially in the education sector, where LLMs can enhance learning but also pose challenges to academic integrity. Detecting AI-generated content (AIGC) is difficult, as existing methods struggle to keep pace with advancements in generation technology. This research proposes a novel approach to AIGC detection in short essays, using digital behavior capture and follow-up questioning to verify text authorship. We executed a controlled experiment as an initial evaluation to test the prototype system. The results obtained show promise in differentiating between user-authored and AI-generated text. The system design and prototype represent valuable contributions for future research in this area. The solution also provides a novel approach to addressing practical challenges posed by LLMs, particularly in maintaining academic integrity in educational settings

Machine Learning Algorithm to Detect Impersonation in an Essay-Based E-Exam

Essay-based E-exams require answers to be written out at some length in an E- learning platform The questions require a response with multiple paragraphs and should be logical and well-structured These type of examinations are increasingly becoming popular in academic institutions of higher learning based on the experience of COVID-19 pandemic Since the exam is mainly done virtually with reduced supervision the risk of impersonation and stolen content from other sources increases Due to this there is need to design cost effective and accurate techniques that are able to detect cheating in an essay based E- exa

Understanding the Keystroke Log:The Effect of Writing Task on Keystroke Features

Keystroke logging is used to automatically record writers' unfolding typing process and to get insight into moments when they struggle composing text. However, it is not clear which and how features from the keystroke log map to higher-level cognitive processes, such as planning and revision. This study aims to investigate the sensitivity of requently used keystroke features across tasks with different cognitive demands. Two keystroke datasets were analyzed: one consisting of a copy task and an email writing task, and one with a larger difference in cognitive demand: a copy task and an academic summary task. The differences across tasks were modeled using Bayesian linear mixed effects models. Posterior distributions were used to compare the strength and direction of the task effects across features and datasets. The results showed that the average of all interkeystroke intervals were found to be stable across tasks. Features related to the time between words and (sub)sentences only differed between the copy and the academic task. Lastly, keystroke features related to the number of words, revisions, and total time, differed across tasks in both datasets. To conclude, our results indicate that the latter features are related to cognitive load or task complexity. In addition, our research shows that keystroke features are sensitive to small differences in the writing tasks at hand

A Practical Model of Student Engagement While Programming

We consider the question of how to predict whether a student is on or off task while working on a computer programming assignment using elapsed time since the last keystroke as the single independent variable. In this paper we report results of an empirical study in which we intermittently prompted CS1 students working on a programming assignment to self-report whether they were engaged in the assignment at that moment. Our regression model derived from the results of the study shows power-law decay in the engagement rate of students with increasing time of keyboard inactivity ranging from a nearly 80% engagement rate after 45 seconds to 30% after 32 minutes of inactivity. We find that students remain engaged in programming for a median of about 8 minutes before going off task, and when they do go off task, they most often return after 1 to 4 minutes of disengagement. Our model has application in estimating the amount of engaged time students take to complete programming assignments, identifying students in need of intervention, and understanding the effects of different engagement behaviors

A Study of Keystroke Data in Two Contexts : Written Language and Programming Language Influence Predictability of Learning Outcomes

We study programming process data from two introductory programming courses. Between the course contexts, the programming languages differ, the teaching approaches differ, and the spoken languages differ. In both courses, students' keystroke data -- timestamps and the pressed keys -- are recorded as students work on programming assignments. We study how the keystroke data differs between the contexts, and whether research on predicting course outcomes using keystroke latencies generalizes to other contexts. Our results show that there are differences between the contexts in terms of frequently used keys, which can be partially explained by the differences between the spoken languages and the programming languages. Further, our results suggest that programming process data that can be collected non-intrusive in-situ can be used for predicting course outcomes in multiple contexts. The predictive power, however, varies between contexts possibly because the frequently used keys differ between programming languages and spoken languages. Thus, context-specific fine-tuning of predictive models may be needed.Peer reviewe