3 research outputs found
Your model is predictive— but is it useful? Theoretical and Empirical Considerations of a New Paradigm for Adaptive Tutoring Evaluation
Classification evaluation metrics are often used to evaluate adaptive tutoring systems— programs that teach and adapt to humans. Unfortunately, it is not clear how intuitive these metrics are for practitioners with little machine learning background. Moreover, our experiments suggest that existing convention for evaluating tutoring systems may lead to suboptimal decisions. We propose the Learner Effort-Outcomes Paradigm (Leopard), a new framework to evaluate adaptive tutoring. We introduce Teal and White, novel automatic metrics that apply Leopard and quantify the amount of effort required to achieve a learning outcome. Our experiments suggest that our metrics are a better alternative for evaluating adaptive tutoring
Recommended from our members
Receiver Operating Characteristic (ROC) Area Under the Curve (AUC): A Diagnostic Measure for Evaluating the Accuracy of Predictors of Education Outcomes
Early Warning Systems (EWS) and Early Warning Indictors (EWI) have recently emerged as an attractive domain for states and school districts interested in predicting student outcomes using data that schools already collect with the intention to better time and tailor interventions. However, current diagnostic measures used across the domain do not consider the dual issues of sensitivity and specificity of predictors, key components for considering accuracy. We apply signal detection theory using Receiver Operating Characteristic (ROC) Area Under the Curve (AUC) analysis adapted from the engineering and medical domains, and using the pROC package in R. Using nationally generalizable data from the Education Longitudinal Study of 2002 (ELS:2002) we provide examples of applying ROC accuracy analysis to a variety of predictors of student outcomes, such as dropping out of high school, college enrollment, and postsecondary STEM degrees and careers.
Keywords: ROC, AUC, Early Warning System, Early Warning Indicator, signal detection theory, dropout, college enrollment, Postsecondary STEM Degree, hard STEM career, soft STEM caree
Prà ctica continuada i feedback automà tic en l'aprenentatge de matemà tiques en lÃnia: un estudi des de la perspectiva de les analÃtiques d'aprenentatge
Aquesta tesi ha adoptat la perspectiva de les analÃtiques d'aprenentatge i s'ha centrat en dues assignatures de matemà tiques en lÃnia de la UOC: Anà lisi matemà tica i EstadÃstica. Hem observat que la qualificació a l'examen final està relacionada amb la realització de qüestionaris i la puntuació obtinguda. També hem comprovat que no presentar-se a l'examen final o no superar-lo és predictible a partir de les qualificacions obtingudes als primers qüestionaris del curs. AixÃ, doncs, s'ha dissenyat i implementat una intervenció docent que permet als estudiants fer els qüestionaris que no havien fet en el termini previst. L'anà lisi d'aquesta intervenció ha permès determinar que ha augmentat la probabilitat de reduir el nombre d'estudiants que no es presenten a l'examen final, cosa que suposa reduir l'abandonament de l'assignatura. Aquesta tesi ens ha permès concloure que mantenir el compromÃs dels estudiants al llarg del curs mitjançant la realització de qüestionaris amb correcció i feedback automà tics en assignatures de matemà tiques en lÃnia ajuda a l'assoliment dels objectius d'aprenentatge.This thesis adopts a learning analytics approach and focuses on two online mathematics courses at the Universitat Oberta de Catalunya (UOC): Calculus and Statistics. Our findings suggest that final exam scores are related to taking quizzes, as well as to quiz scores. Specifically, we show that not taking or succeeding in the final exam can be predicted from students' scores on the first few quizzes of the academic year. A teaching intervention was designed and implemented to allow students to take any of the quizzes that they had not submitted before the original deadline. Analysing the effectiveness of this intervention, we have found that it improves students' chances of taking the final exam, and therefore reduces student drop-out in the Statistics course. This doctoral thesis has allowed us to conclude that, for online mathematics courses, being engaged throughout the course by taking quizzes with automatic correction and feedback helps students achieve their learning goals.Esta tesis ha adoptado la perspectiva de las analÃticas de aprendizaje y se ha centrado en dos asignaturas de matemáticas en lÃnea de la UOC: Análisis matemático y EstadÃstica. Hemos observado que la calificación en el examen final está relacionada con la realización de cuestionarios y la puntuación obtenida. Asimismo, hemos comprobado que no presentarse al examen final o no superarlo es predecible a partir de las calificaciones obtenidas en los primeros cuestionarios del curso. AsÃ, se ha diseñado e implementado una intervención docente que permite a los estudiantes hacer los cuestionarios que no habÃan realizado en el plazo previsto. El análisis de esta intervención ha permitido determinar que ha aumentado la probabilidad de reducir el número de estudiantes que no se presentan al examen final, hecho que supone reducir el abandono de la asignatura. Esta tesis nos ha permitido concluir que mantener el compromiso de los estudiantes a lo largo del curso mediante la realización de cuestionarios con corrección y feedback automáticos en asignaturas de matemáticas en lÃnea ayuda a alcanzar los objetivos de aprendizaje