Qualitative, quantitative, and data mining methods for analyzing log data to characterize students' learning strategies and behaviors [discussant]

This symposium addresses how different classes of research methods, all based upon the use of log data from educational software, can facilitate the analysis of students’ learning strategies and behaviors. To this end, four multi-method programs of research are discussed, including the use of qualitative, quantitative-statistical, quantitative-modeling, and educational data mining methods. The symposium presents evidence regarding the applicability of each type of method to research questions of different grain sizes, and provides several examples of how these methods can be used in concert to facilitate our understanding of learning processes, learning strategies, and behaviors related to motivation, meta-cognition, and engagement

Carelessness and Affect in an Intelligent Tutoring System for Mathematics

We investigate the relationship between students’ affect and their frequency of careless errors while using an Intelligent Tutoring System for middle school mathematics. A student is said to have committed a careless error when the student’s answer is wrong despite knowing the skill required to provide the correct answer. We operationalize the probability that an error is careless through the use of an automated detector, developed using educational data mining, which infers the probability that an error involves carelessness rather than not knowing the relevant skill. This detector is then applied to log data produced by high-school students in the Philippines using a Cognitive Tutor for scatterplots. We study the relationship between carelessness and affect, triangulating between the detector of carelessness and field observations of affect. Surprisingly, we find that carelessness is common among students who frequently experience engaged concentration. This finding implies that a highly engaged student may paradoxically become overconfident or impulsive, leading to more careless errors. In contrast, students displaying confusion or boredom make fewer careless errors. Further analysis over time suggests that confused and bored students have lower learning overall. Thus, their mistakes appear to stem from a genuine lack of knowledge rather than carelessness

Intelligent tutoring in virtual reality for highly dynamic pedestrian safety training

This thesis presents the design, implementation, and evaluation of an Intelligent Tutoring System (ITS) with a Virtual Reality (VR) interface for child pedestrian safety training. This system enables children to train practical skills in a safe and realistic virtual environment without the time and space dependencies of traditional roadside training. This system also employs Domain and Student Modelling techniques to analyze user data during training automatically and to provide appropriate instructions and feedback. Thus, the traditional requirement of constant monitoring from teaching personnel is greatly reduced. Compared to previous work, especially the second aspect is a principal novelty for this domain. To achieve this, a novel Domain and Student Modeling method was developed in addition to a modular and extensible virtual environment for the target domain. While the Domain and Student Modeling framework is designed to handle the highly dynamic nature of training in traffic and the ill-defined characteristics of pedestrian tasks, the modular virtual environment supports different interaction methods and a simple and efficient way to create and adapt exercises. The thesis is complemented by two user studies with elementary school children. These studies testify great overall user acceptance and the system’s potential for improving key pedestrian skills through autonomous learning. Last but not least, the thesis presents experiments with different forms of VR input and provides directions for future work.Diese Arbeit behandelt den Entwurf, die Implementierung sowie die Evaluierung eines intelligenten Tutorensystems (ITS) mit einer Virtual Reality (VR) basierten Benutzeroberfläche zum Zwecke von Verkehrssicherheitstraining für Kinder. Dieses System ermöglicht es Kindern praktische Fähigkeiten in einer sicheren und realistischen Umgebung zu trainieren, ohne den örtlichen und zeitlichen Abhängigkeiten des traditionellen, straßenseitigen Trainings unterworfen zu sein. Dieses System macht außerdem von Domain und Student Modelling Techniken gebrauch, um Nutzerdaten während des Trainings zu analysieren und daraufhin automatisiert geeignete Instruktionen und Rückmeldung zu generieren. Dadurch kann die bisher erforderliche, ständige Überwachung durch Lehrpersonal drastisch reduziert werden. Verglichen mit bisherigen Lösungen ist insbesondere der zweite Aspekt eine grundlegende Neuheit für diesen Bereich. Um dies zu erreichen wurde ein neuartiges Framework für Domain und Student Modelling entwickelt, sowie eine modulare und erweiterbare virtuelle Umgebung für diese Art von Training. Während das Domain und Student Modelling Framework so entworfen wurde, um mit der hohen Dynamik des Straßenverkehrs sowie den vage definierten Fußgängeraufgaben zurecht zu kommen, unterstützt die modulare Umgebung unterschiedliche Eingabeformen sowie eine unkomplizierte und effiziente Methode, um Übungen zu erstellen und anzupassen. Die Arbeit beinhaltet außerdem zwei Nutzerstudien mit Grundschulkindern. Diese Studien belegen dem System eine hohe Benutzerakzeptanz und stellt das Potenzial des Systems heraus, wichtige Fähigkeiten für Fußgängersicherheit durch autodidaktisches Training zu verbessern. Nicht zuletzt beschreibt die Arbeit Experimente mit verschiedenen Formen von VR Eingaben und zeigt die Richtung für zukünftige Arbeit auf

Intelligent Tutoring System Authoring Tools for Non-Programmers

An intelligent tutoring system (ITS) is a software application that tries to replicate the performance of a human tutor by supporting the theory of learning by doing . ITSs have been shown to improve the performance of a student in wide range of domains. Despite their benefits, ITSs have not seen widespread use due to the complexity involved in their development. Developing an ITS from scratch requires expertise in several fields including computer science, cognitive psychology and artificial intelligence. In order to decrease the skill threshold required to build ITSs, several authoring tools have been developed. In this thesis, I document several contributions to the field of intelligent tutoring in the form of extensions to an existing ITS authoring tool, research studies on authoring tool paradigms and the design of authoring tools for non-programmers in two complex domains - natural language processing and 3D game environments. The Extensible Problem Specific Tutor (xPST) is an authoring tool that helps rapidly develop model-tracing like tutors on existing interfaces such as webpages. xPST\u27s language was made more expressive with the introduction of new checktypes required for answer checking in problems belonging to domains such as geometry and statistics. A web-based authoring (WAT) tool was developed for the purpose of tutor management and deployment and to promote non-programmer authoring of ITSs. The WAT was used in a comparison study between two authoring tool paradigms - GUI based and text based, in two different problem domains - statistics and geometry. User-programming of natural language processing (NLP) in ITSs is not common with authoring toolkits. Existing NLP techniques do not offer sufficient power to non-programmers and the NLP is left to expert developers or machine learning algorithms. We attempted to address this challenge by developing a domain-independent authoring tool, ConceptGrid that is intended to help non-programmers develop ITSs that perform natural language processing. ConceptGrid has been integrated into xPST. When templates created using ConceptGrid were tested, they approached the accuracy of human instructors in scoring student responses. 3D game environments belong to another domain for which authoring tools are uncommon. Authoring game-based tutors is challenging due to the inherent domain complexity and dynamic nature of the environment. We attempt to address this challenge through the design of authoring tool that is intended to help non-programmers develop game-based ITSs

Adaptive intelligent tutoring for teaching modern standard Arabic

A thesis submitted to the University of Bedfordshire, in partial fulfilment of the requirements for the degree of Doctor of PhilosophyThe aim of this PhD thesis is to develop a framework for adaptive intelligent tutoring systems (ITS) in the domain of Modern Standard Arabic language. This framework will comprise of a new approach to using a fuzzy inference mechanism and generic rules in guiding the learning process. In addition, the framework will demonstrate another contribution in which the system can be adapted to be used in the teaching of different languages. A prototype system will be developed to demonstrate these features. This system is targeted at adult English-speaking casual learners with no pre-knowledge of the Arabic language. It will consist of two parts: an ITS for learners to use and a teachers‘ tool for configuring and customising the teaching rules and artificial intelligence components among other configuration operations. The system also provides a diverse teaching-strategies‘ environment based on multiple instructional strategies. This approach is based on general rules that provide means to a reconfigurable prediction. The ITS determines the learner‘s learning characteristics using multiple fuzzy inferences. It has a reconfigurable design that can be altered by the teacher at runtime via a teacher-interface. A framework for an independent domain (i.e. pluggable-domain) for foreign language tutoring systems is introduced in this research. This approach allows the system to adapt to the teaching of a different language with little changes required. Such a feature has the advantages of reducing the time and cost required for building intelligent language tutoring systems. To evaluate the proposed system, two experiments are conducted with two versions of the software: the ITS and a cut down version with no artificial intelligence components. The learners used the ITS had shown an increase in scores between the post-test and the pre-test with learning gain of 35% compared to 25% of the learners from the cut down version

New measurement paradigms

This collection of New Measurement Paradigms papers represents a snapshot of the variety of measurement methods in use at the time of writing across several projects funded by the National Science Foundation (US) through its REESE and DR K–12 programs. All of the projects are developing and testing intelligent learning environments that seek to carefully measure and promote student learning, and the purpose of this collection of papers is to describe and illustrate the use of several measurement methods employed to achieve this. The papers are deliberately short because they are designed to introduce the methods in use and not to be a textbook chapter on each method. The New Measurement Paradigms collection is designed to serve as a reference point for researchers who are working in projects that are creating e-learning environments in which there is a need to make judgments about students’ levels of knowledge and skills, or for those interested in this but who have not yet delved into these methods

Discovery with Models: A Case Study on Carelessness in Computer-based Science Inquiry

In recent years, an increasing number of analyses in Learning Analytics and Educational Data Mining (EDM) have adopted a "Discovery with Models" approach, where an existing model is used as a key component in a new EDM/analytics analysis. This article presents a theoretical discussion on the emergence of discovery with models, its potential to enhance research on learning and learners, and key lessons learned in how discovery with models can be conducted validly and effectively. We illustrate these issues through discussion of a case study where discovery with models was used to investigate a form of disengaged behavior, i.e., carelessness, in the context of middle school computer-based science inquiry. This behavior has been acknowledged as a problem in education as early as the 1920s. With the increasing use of high-stakes testing, the cost of student carelessness can be higher. For instance, within computer-based learning environments careless errors can result in reduced educational effectiveness, with students continuing to receive material they have already mastered. Despite the importance of this problem, it has received minimal research attention, in part due to difficulties in operationalizing carelessness as a construct. Building from theory on carelessness and a Bayesian framework for knowledge modeling, we use machine-learned detectors to predict carelessness within authentic use of a computer-based learning environment. We then use a discovery with models approach to link these validated carelessness measures to survey data, to study the correlations between the prevalence of carelessness and student goal orientation. The second construct, carelessness, refers to incorrect answers given by a student on material that the student should be able to answer correctly Rodriguez-Fornells & Maydeu-Olivares, 2000). The application of discovery with models involves two main phases. First, a model of a construct is developed using machine learning or knowledge engineering techniques, and is then validated, as discussed below. Second, this validated model is applied to data and used as a component in another analysis: For example, for identifying outliers through model predictions; examining which variables best predict the modeled construct; finding relationships between the construct and other variables using correlations, predictions, associations rules, causal relationships or other methods; or studying the contexts where the construct occurs, including its prevalence across domains, systems, or populations. For example, in One essential question to pose prior to a discovery with model analysis is whether the model adopted is valid, both overall, and for the specific situation in which it is being used. Ideally, a model should be validated using an approach such as cross-validation, where the model is repeatedly trained on one portion of the data and tested on a different portion, with model predictions compared to appropriate external measures, for example assessments made by humans with acceptably high inter-rater reliability, such as field observations of student behavior for gaming the system (cf. Even after validating in this fashion, validity should be re-considered if the model is used for a substantially different population or context than was used when developing the model.. An alternative approach is to use a simpler knowledge-engineered definition, rationally deriving a function/rule that is then applied to the data. In this case, the model can be inferred to have face validity. However, knowledge-engineered models often DISCOVERY WITH MODELS: A CASE STUDY ON CARELESSNESS 6 produce different results than machine learning-based models, for example in the case of gaming the system. Research studying whether student or content is a better predictor of gaming the system identified different results, depending on which model was applied (cf. Baker, 2007a