758 research outputs found
Analysis of Student Behaviour in Habitable Worlds Using Continuous Representation Visualization
We introduce a novel approach to visualizing temporal clickstream behaviour
in the context of a degree-satisfying online course, Habitable Worlds, offered
through Arizona State University. The current practice for visualizing
behaviour within a digital learning environment has been to generate plots
based on hand engineered or coded features using domain knowledge. While this
approach has been effective in relating behaviour to known phenomena, features
crafted from domain knowledge are not likely well suited to make unfamiliar
phenomena salient and thus can preclude discovery. We introduce a methodology
for organically surfacing behavioural regularities from clickstream data,
conducting an expert in-the-loop hyperparameter search, and identifying
anticipated as well as newly discovered patterns of behaviour. While these
visualization techniques have been used before in the broader machine learning
community to better understand neural networks and relationships between word
vectors, we apply them to online behavioural learner data and go a step
further; exploring the impact of the parameters of the model on producing
tangible, non-trivial observations of behaviour that are suggestive of
pedagogical improvement to the course designers and instructors. The
methodology introduced in this paper led to an improved understanding of
passing and non-passing student behaviour in the course and is widely
applicable to other datasets of clickstream activity where investigators and
stakeholders wish to organically surface principal patterns of behaviour
Learning gain differences between ChatGPT and human tutor generated algebra hints
Large Language Models (LLMs), such as ChatGPT, are quickly advancing AI to
the frontiers of practical consumer use and leading industries to re-evaluate
how they allocate resources for content production. Authoring of open
educational resources and hint content within adaptive tutoring systems is
labor intensive. Should LLMs like ChatGPT produce educational content on par
with human-authored content, the implications would be significant for further
scaling of computer tutoring system approaches. In this paper, we conduct the
first learning gain evaluation of ChatGPT by comparing the efficacy of its
hints with hints authored by human tutors with 77 participants across two
algebra topic areas, Elementary Algebra and Intermediate Algebra. We find that
70% of hints produced by ChatGPT passed our manual quality checks and that both
human and ChatGPT conditions produced positive learning gains. However, gains
were only statistically significant for human tutor created hints. Learning
gains from human-created hints were substantially and statistically
significantly higher than ChatGPT hints in both topic areas, though ChatGPT
participants in the Intermediate Algebra experiment were near ceiling and not
even with the control at pre-test. We discuss the limitations of our study and
suggest several future directions for the field. Problem and hint content used
in the experiment is provided for replicability
Ensembling predictions of student post-test scores for an intelligent tutoring system.
________________________________________________________________________ Over the last few decades, there have been a rich variety of approaches towards modeling student knowledge and skill within interactive learning environments. There have recently been several empirical comparisons as to which types of student models are better at predicting future performance, both within and outside of the interactive learning environment. A recent paper (Baker et al., in press) considers whether ensembling can produce better prediction than individual models, when ensembling is performed at the level of predictions of performance within the tutor. However, better performance was not achieved for predicting the post-test. In this paper, we investigate ensembling at the post-test level, to see if this approach can produce better prediction of post-test scores within the context of a Cognitive Tutor for Genetics. We find no improvement for ensembling over the best individual models and we consider possible explanations for this finding, including the limited size of the data set
Learner Modeling for Integration Skills
Complex skill mastery requires not only acquiring individual basic component skills, but also practicing integrating such basic skills. However, traditional approaches to knowledge modeling, such as Bayesian knowledge tracing, only trace knowledge of each decomposed basic component skill. This risks early assertion of mastery or ineffective remediation failing to address skill integration. We introduce a novel integration-level approach to model learners' knowledge and provide fine-grained diagnosis: a Bayesian network based on a new kind of knowledge graph with progressive integration skills. We assess the value of such a model from multifaceted aspects: performance prediction, parameter plausibility, expected instructional effectiveness, and real-world recommendation helpfulness. Our experiments based on a Java programming tutor show that proposed model significantly improves two popular multiple-skill knowledge tracing models on all these four aspects
- …