The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

Detecting Mistakes in CPR Training with Multimodal Data and Neural Networks

This study investigated to what extent multimodal data can be used to detect mistakes during Cardiopulmonary Resuscitation (CPR) training. We complemented the Laerdal QCPR ResusciAnne manikin with the Multimodal Tutor for CPR, a multi-sensor system consisting of a Microsoft Kinect for tracking body position and a Myo armband for collecting electromyogram information. We collected multimodal data from 11 medical students, each of them performing two sessions of two-minute chest compressions (CCs). We gathered in total 5254 CCs that were all labelled according to five performance indicators, corresponding to common CPR training mistakes. Three out of five indicators, CC rate, CC depth and CC release, were assessed automatically by the ReusciAnne manikin. The remaining two, related to arms and body position, were annotated manually by the research team. We trained five neural networks for classifying each of the five indicators. The results of the experiment show that multimodal data can provide accurate mistake detection as compared to the ResusciAnne manikin baseline. We also show that the Multimodal Tutor for CPR can detect additional CPR training mistakes such as the correct use of arms and body weight. Thus far, these mistakes were identified only by human instructors. Finally, to investigate user feedback in the future implementations of the Multimodal Tutor for CPR, we conducted a questionnaire to collect valuable feedback aspects of CPR training. (DIPF/Orig.

Capturing and Scaffolding the Complexities of Self-Regulation During Game-Based Learning

Game-based learning environments (GBLEs) can offer students with engaging interactive instructional materials while also providing a research platform to investigate the dynamics and intricacies of effective self-regulated learning (SRL). Past research has indicated learners are often unable to monitor and regulate their cognitive and metacognitive processes within GBLEs accurately and effectively on their own due mostly to the open-ended nature of these environments. The future design and development of GBLEs and embedded scaffolds, therefore, require a better understanding of the discrepancies between the affordances of GBLEs and the required use of SRL. Specifically, how to incorporate interdisciplinary theories and concepts outside of traditional educational, learning, and psychological sciences literature, how to utilize process data to measure SRL processes during interactions with instructional materials accounting for the dynamics of leaners\u27 SRL, and how to improve SRL-driven scaffolds to be individualized and adaptive based on the level of agency GBLEs provide. Across four studies, this dissertation investigates learners\u27 SRL while they learn about microbiology using CRYSTAL ISLAND, a GBLE, building upon each other by enhancing the type of data collected, analytical methodologies used, and applied theoretical models and theories. Specifically, this dissertation utilizes a combination of traditional statistical approaches (i.e., linear regression models), non-linear statistical approaches (i.e., growth modeling), and non-linear dynamical theory (NDST) approaches (aRQA) with process trace data to contribute to the field\u27s current understanding of the dynamics and complexities of SRL. Furthermore, this dissertation examines how limited agency can act as an implicit scaffold during game-based learning to promote the use of SRL processes and increase learning outcomes