5 research outputs found
Your model is predictive— but is it useful? Theoretical and Empirical Considerations of a New Paradigm for Adaptive Tutoring Evaluation
Classification evaluation metrics are often used to evaluate adaptive tutoring systems— programs that teach and adapt to humans. Unfortunately, it is not clear how intuitive these metrics are for practitioners with little machine learning background. Moreover, our experiments suggest that existing convention for evaluating tutoring systems may lead to suboptimal decisions. We propose the Learner Effort-Outcomes Paradigm (Leopard), a new framework to evaluate adaptive tutoring. We introduce Teal and White, novel automatic metrics that apply Leopard and quantify the amount of effort required to achieve a learning outcome. Our experiments suggest that our metrics are a better alternative for evaluating adaptive tutoring
Beyond A/B Testing: Sequential Randomization for Developing Interventions in Scaled Digital Learning Environments
Randomized experiments ensure robust causal inference that are critical to
effective learning analytics research and practice. However, traditional
randomized experiments, like A/B tests, are limiting in large scale digital
learning environments. While traditional experiments can accurately compare two
treatment options, they are less able to inform how to adapt interventions to
continually meet learners' diverse needs. In this work, we introduce a trial
design for developing adaptive interventions in scaled digital learning
environments -- the sequential randomized trial (SRT). With the goal of
improving learner experience and developing interventions that benefit all
learners at all times, SRTs inform how to sequence, time, and personalize
interventions. In this paper, we provide an overview of SRTs, and we illustrate
the advantages they hold compared to traditional experiments. We describe a
novel SRT run in a large scale data science MOOC. The trial results
contextualize how learner engagement can be addressed through inclusive
culturally targeted reminder emails. We also provide practical advice for
researchers who aim to run their own SRTs to develop adaptive interventions in
scaled digital learning environments
Personalized Stopping Rules in Bayesian Adaptive Mastery Assessment
We propose a new model to assess the mastery level of a given skill
efficiently. The model, called Bayesian Adaptive Mastery Assessment (BAMA),
uses information on the accuracy and the response time of the answers given and
infers the mastery at every step of the assessment. BAMA balances the length of
the assessment and the certainty of the mastery inference by employing a
Bayesian decision-theoretic framework adapted to each student. All these
properties contribute to a novel approach in assessment models for intelligent
learning systems. The purpose of this research is to explore the properties of
BAMA and evaluate its performance concerning the number of questions
administered and the accuracy of the final mastery estimates across different
students. We simulate student performances and establish that the model
converges with low variance and high efficiency leading to shorter assessment
duration for all students. Considering the experimental results, we expect our
approach to avoid the issue of over-practicing and under-practicing and
facilitate the development of Learning Analytics tools to support the tutors in
the evaluation of learning effects and instructional decision making.Comment: 12 page