Creating Systems and Applying Large-Scale Methods to Improve Student Remediation in Online Tutoring Systems in Real-time and at Scale

A common problem shared amongst online tutoring systems is the time-consuming nature of content creation. It has been estimated that an hour of online instruction can take up to 100-300 hours to create. Several systems have created tools to expedite content creation, such as the Cognitive Tutors Authoring Tool (CTAT) and the ASSISTments builder. Although these tools make content creation more efficient, they all still depend on the efforts of a content creator and/or past historical. These tools do not take full advantage of the power of the crowd. These issues and challenges faced by online tutoring systems provide an ideal environment to implement a solution using crowdsourcing. I created the PeerASSIST system to provide a solution to the challenges faced with tutoring content creation. PeerASSIST crowdsources the work students have done on problems inside the ASSISTments online tutoring system and redistributes that work as a form of tutoring to their peers, who are in need of assistance. Multi-objective multi-armed bandit algorithms are used to distribute student work, which balance exploring which work is good and exploiting the best currently known work. These policies are customized to run in a real-world environment with multiple asynchronous reward functions and an infinite number of actions. Inspired by major companies such as Google, Facebook, and Bing, PeerASSIST is also designed as a platform for simultaneous online experimentation in real-time and at scale. Currently over 600 teachers (grades K-12) are requiring students to show their work. Over 300,000 instances of student work have been collected from over 18,000 students across 28,000 problems. From the student work collected, 2,000 instances have been redistributed to over 550 students who needed help over the past few months. I conducted a randomized controlled experiment to evaluate the effectiveness of PeerASSIST on student performance. Other contributions include representing learning maps as Bayesian networks to model student performance, creating a machine-learning algorithm to derive student incorrect processes from their incorrect answer and the inputs of the problem, and applying Bayesian hypothesis testing to A/B experiments. We showed that learning maps can be simplified without practical loss of accuracy and that time series data is necessary to simplify learning maps if the static data is highly correlated. I also created several interventions to evaluate the effectiveness of the buggy messages generated from the machine-learned incorrect processes. The null results of these experiments demonstrate the difficulty of creating a successful tutoring and suggest that other methods of tutoring content creation (i.e. PeerASSIST) should be explored

The Hidden Value of Common Wrong Answers

In this report, we investigate the value of common wrong answers. By introducing common wrong answers into model building, can we achieve better models in predicting next problem correctness? To answer our question we build two tabling models. The first model is our control model. This model makes predictions based on three types of student responses, correct responses, hint responses and wrong answer responses. The second model is our experiment model, named Interesting Common Wrong Answers Model (ICWAs Model). The ICWAs model extends upon the control model by giving each common wrong answer its own prediction. We compared the results from both models and verified that common wrong answers do not bring improvements to predicting next problem correctness

Hint generation in programming tutors

Programming is increasingly recognized as a useful and important skill. Online programming courses that have appeared in the past decade have proven extremely popular with a wide audience. Learning in such courses is however not as effective as working directly with a teacher, who can provide students with immediate relevant feedback. The field of intelligent tutoring systems seeks to provide such feedback automatically. Traditionally, tutors have depended on a domain model defined by the teacher in advance. Creating such a model is a difficult task that requires a lot of knowledgeengineering effort, especially in complex domains such as programming. A potential solution to this problem is to use data-driven methods. The idea is to build the domain model by observing how students have solved an exercise in the past. New students can then be given feedback that directs them along successful solution paths. Implementing this approach is particularly challenging for programming domains, since the only directly observable student actions are not easily interpretable. We present two novel approaches to creating a domain model for programming exercises in a data-driven fashion. The first approach models programming as a sequence of textual rewrites, and learns rewrite rules for transforming programs. With these rules new student-submitted programs can be automatically debugged. The second approach uses structural patterns in programs’ abstract syntax trees to learn rules for classifying submissions as correct or incorrect. These rules can be used to find erroneous parts of an incorrect program. Both models support automatic hint generation. We have implemented an online application for learning programming and used it to evaluate both approaches. Results indicate that hints generated using either approach have a positive effect on student performance

Un modèle pour la génération d'indices par une plateforme de tuteurs par traçage de modèle

La présente thèse décrit des travaux de recherche effectués dans le domaine des systèmes tutoriels intelligents (STI). Plus particulièrement, elle s'intéresse aux tuteurs par traçage de modèle (MTT). Les MTTs ont montré leur efficacité pour le tutorat de la résolution de tâches bien définies. Par contre, les interventions pédagogiques qu'ils produisent doivent être incluses, par l'auteur du tuteur, dans le modèle de la tâche enseignée. La recherche effectuée répond à cette limite en proposant des méthodes et algorithmes permettant la génération automatique d'interventions pédagogiques. Une méthode a été développée afin de permettre à la plateforme Astus de générer des indices par rapport à la prochaine étape en examinant le contenu du modèle de la tâche enseignée. De plus, un algorithme a été conçu afin de diagnostiquer les erreurs des apprenants en fonction des actions hors trace qu'ils commettent. Ce diagnostic permet à Astus d'offrir une rétroaction par rapport aux erreurs sans que l'auteur du tuteur ait à explicitement modéliser les erreurs. Cinq expérimentations ont été effectuées lors de cours enseignés au département d'informatique de l'Université de Sherbrooke afin de valider de façon empirique les interventions générées par Astus. Le résultat de ces expérimentations montre que 1) il est possible de générer des indices par rapport à la prochaine étape qui sont aussi efficaces et aussi appréciés que ceux conçus par un enseignant et que 2) la plateforme Astus est en mesure de diagnostiquer un grand nombre d'actions hors trace des apprenants afin de fournir une rétroaction par rapport aux erreurs

A methodology for evaluating intelligent tutoring systems

DissertationThis dissertation proposes a generic methodology for evaluating intelligent tutoring systems (ITSs), and applies it to the evaluation of the SQL-Tutor, an ITS for the database language SQL. An examination of the historical development, theory and architecture of intelligent tutoring systems, as well as the theory, architecture and behaviour of the SQL-Tutor sets the context for this study. The characteristics and criteria for evaluating computer-aided instruction (CAl) systems are considered as a background to an in-depth investigation of the characteristics and criteria appropriate for evaluating ITSs. These criteria are categorised along internal and external dimensions with the internal dimension focusing on the intrinsic features and behavioural aspects of ITSs, and the external dimension focusing on its educational impact. Several issues surrounding the evaluation of ITSs namely, approaches, methods, techniques and principles are examined, and integrated within a framework for assessing the added value of ITS technology for instructional purposes.Educational StudiesM. Sc. (Information Systems

Investigating the Effectiveness of Problem Templates on Learning in Intelligent Tutoring Systems

Deliberate practice within a coached environment is required for skill acquisition and mastery. Intelligent Tutoring Systems (ITSs) provide such an environment. A goal in ITS development is to find means to maximise effective learning. This provides the motivation for the project presented. This paper proposes the notion of problem templates. These mental constructs extend the idea of memory templates, and allow experts in a domain to store vast amounts of domain-specific information that are easily accessible when faced with a problem. This research aims to examine the validity of such a construct and investigate its role in regards to effective learning within ITSs. After extensive background research, an evaluation study was performed at the University of Canterbury. Physical representations of problem templates were formed in Structured Query Language (SQL). These were used to model students, select problems, and provide customised feedback in the experimental version of SQLTutor, an Intelligent Tutoring System. The control group used the original version of SQL-Tutor where pedagogical (problem selection and feedback) and modelling decisions were based on constraints. Preliminary results show that such a construct could exist; furthermore, it could be used to help students attain high levels of expertise within a domain. Students using template based ITS showed high levels of learning within short periods of time. The author suggests further evaluation studies to investigate the extent and detail of its effect on learning

Widening the Knowledge Acquisition Bottleneck for Intelligent Tutoring Systems

Empirical studies have shown that Intelligent Tutoring Systems (ITS) are effective tools for education. However, developing an ITS is a labour-intensive and time-consuming process. A major share of the development effort is devoted to acquiring the domain knowledge that accounts for the intelligence of the system. The goal of this research is to reduce the knowledge acquisition bottleneck and enable domain experts to build the domain model required for an ITS. In pursuit of this goal an authoring system capable of producing a domain model with the assistance of a domain expert was developed. Unlike previous authoring systems, this system (named CAS) has the ability to acquire knowledge for non-procedural as well as procedural tasks. CAS was developed to generate the knowledge required for constraint-based tutoring systems, reducing the effort as well as the amount of expertise in knowledge engineering and programming required. Constraint-based modelling is a student modelling technique that assists in somewhat easing the knowledge acquisition bottleneck due to the abstract representation. CAS expects the domain expert to provide an ontology of the domain, example problems and their solutions. It uses machine learning techniques to reason with the information provided by the domain expert for generating a domain model. A series of evaluation studies of this research produced promising results. The initial evaluation revealed that the task of composing an ontology of the domain assisted with the manual composition of a domain model. The second study showed that CAS was effective in generating constraints for the three vastly different domains of database modelling, data normalisation and fraction addition. The final study demonstrated that CAS was also effective in generating constraints when assisted by novice ITS authors, producing constraint sets that were over 90% complete

An intelligent teaching system for database modeling.

Database (DB) modelling, like other analysis and design tasks, can only be learnt through extensive practice. Conventionally, DB modelling is taught in a classroom environment where the instructor demonstrates the task using typical examples and students practise modelling in labs or tutorials. Although one-to-one human tutoring is the most effective mode of teaching, there will never be sufficient resources to provide individualised attention to each and every student. However, Intelligent Teaching Systems (ITS) offer bright prospects to fulfilling the goal of providing individualised pedagogical sessions to all students. Studies have shown that ITSs with problem-solving environments are ideal tools for enhancing learning in domains where extensive practice is essential. This thesis describes the design, implementation and evaluation of an ITS named KERMIT, developed for the popular database modelling technique, Entity Relationship (ER) modelling. KERMIT, the Knowledge-based Entity Relationship Modelling Intelligent Tutor, is developed as a problem-solving environment in which students can practice their ER modelling skills with the individualised assistance of the system. KERMIT presents a description of a scenario for which the student models a database using ER modelling constructs. The student can ask for guidance from the system during any stage of the problem solving process, and KERMIT evaluates the solution and presents feedback on its errors. The system adapts to each individual student by providing individualised hint messages and selecting new problems that best suit the student. The effectiveness of KERMIT was tested by three evaluations. The first was a think-aloud study to gain first-hand experience of the student's perception of the system. The second study, conducted as a classroom experiment, yielded some positive results, considering the time limitations and the instabilities of the system. The third evaluation, a similar classroom experiment, clearly demonstrated the effectiveness of KERMIT as a teaching system. Students were divided into an experimental group that interacted with KERMIT and a control group that used a conventional drawing tool to practice ER modelling. Both group's learning was monitored by pre- and post-tests, and a questionnaire recorded their perception of the system. The results of the study showed that students using KERMIT showed a significantly higher gain in their post-test. Their responses to the questionnaire reaffirmed their positive perception of KERMIT. The usefulness of feedback from the system and the amount learnt from the system was also on a significantly higher scale. Their free-form comments were also very positive