Dyadic joint visual attention interaction in face-to-face collaborative problem-solving at K-12 Maths Education: A Multimodal Approach

Collaborative problem-solving (CPS) is an essential skill in the workplace in the 21st century, but the assessment and support of the CPS process with scientifically objective evidence are challenging. This research aims to understand in-class CPS interaction by investigating the change of a dyad's cognitive engagement during a mathematics lesson. Here, we propose a multimodal evaluation of joint visual attention (JVA) based on eye gazes and eye blinks data as non-verbal indicators of dyadic cognitive engagement. Our results indicate that this multimodal approach can bring more insights into students' CPS process than unimodal evaluations of JVA in temporal analysis. This study contributes to the field by demonstrating the value of nonverbal multimodal JVA temporal analysis in CPS assessment and the utility of eye physiological data in improving the interpretation of dyadic cognitive engagement. Moreover, a method is proposed for capturing gaze convergence by considering eye fixations and the overlapping time between two eye gazes. We conclude the paper with our preliminary findings from a pilot study investigating the proposed approach in a real-world teaching context

The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

DESIGN AND EXPLORATION OF NEW MODELS FOR SECURITY AND PRIVACY-SENSITIVE COLLABORATION SYSTEMS

Collaboration has been an area of interest in many domains including education, research, healthcare supply chain, Internet of things, and music etc. It enhances problem solving through expertise sharing, ideas sharing, learning and resource sharing, and improved decision making. To address the limitations in the existing literature, this dissertation presents a design science artifact and a conceptual model for collaborative environment. The first artifact is a blockchain based collaborative information exchange system that utilizes blockchain technology and semi-automated ontology mappings to enable secure and interoperable health information exchange among different health care institutions. The conceptual model proposed in this dissertation explores the factors that influences professionals continued use of video- conferencing applications. The conceptual model investigates the role the perceived risks and benefits play in influencing professionals’ attitude towards VC apps and consequently its active and automatic use