Instruments for visualization of self, co, and socially shared regulation of learning using multimodal analytics:a systematic review

Abstract. This thesis presents a systematic literature review in the intersection of multimodal learning analytics, regulation theories of learning, and visual analytics literature of the last decade (2011- 2021). This review is to collect existing research-based instruments designed to visualize Self-Regulation of Learning (SRL), Co-Regulation of learning (CoRL), and Socially Shared Regulation of learning (SSRL) using dashboards and multimodal data. The inclusion and exclusion criteria used in this review addressed two main aims. First, to distil settings, instruments, constructs, and audiences. Second, to identify visualization used for targets (i.e., cognition, motivation, and emotion), phases (i.e., forethought, performance, and reflection), and types of regulation (i.e., SRL, CoRL, and SSRL). By following the Preferred Reporting Items for Systematic Reviews and MetaAnalyses (PRISMA) guidelines, this thesis included 23 peer-reviewed articles out of 383 articles retrieved from 5 different databases searched in April 2021. The main findings from this literature review are (a) the included articles used theoretical grounding of SRL in all articles while CoRL is used only in 3 articles and SSRL only in 2 articles; (b) most articles used both teachers and students as the audience for visual feedback and operated in online learning settings; (c) selected articles focused mainly on visualizing cognition and motivation (17 articles each) as targets of regulation, while emotion as the target was applied only in 6 articles; (d) The performance phase was common to most of the articles and used various visualizations followed by reflection and forethought phases respectively. Simple visualizations, i.e., progress bar chart, line chart, color coding, are used more frequently than bubble chart, stacked column chart, funnel chart, heat maps, and Sankey diagram. Most of the dashboard instruments identified in the review are still improving their designs. Therefore, the results of this review should be put into the context of future studies to be utilized by researchers and teachers in recognizing the missing targets and phases of SRL, CoRL, and SSRL in visualized feedback. Addressing these could also assist them in giving timely feedback on students’ learning strategies to improve their regulatory skills

Supporting STEM knowledge and skills in engineering education – PELARS project

In this paper we present our proposal for improving education with hands-on, project-based and experimental scenarios for engineering students with the use of learning analytics. We accomplish this through teacher and learner engagement, user studies and evaluated trials, performed at UCV (University of Craiova, Romania) and DTU (Technical University of Denmark). The PELARS project (Practice-based Experiential Learning Analytics Research And Support) provides technological tools and ICT-based methods for collecting activity data (moving image-based and embedded sensing) for learning analytics (data-mining and reasoning) of practice-based and experiential STEM. This data is used to create analytics support tools for teachers, learners and administrators, providing frameworks for evidence-based curriculum design and learning systems. The PELARS project creates behavioral recording inputs, proving a new learning analytic that is scalable in application, and bridge qualitative and quantitative methods through reasoning and feedback from input data. The project serves to better understand learners' knowledge in physical activities in laboratory and workshop environments, as well as informal learning scenarios. PELARS traces and helps assess learner progress through technology enhancement, in novel ways building upon current research. The project results in learning analytics tools for practice-based STEM learning that are appropriate for real-world learning environments

Collocated Collaboration Analytics: Principles and Dilemmas for Mining Multimodal Interaction Data

© 2019, Copyright © 2017 Taylor & Francis Group, LLC. Learning to collaborate effectively requires practice, awareness of group dynamics, and reflection; often it benefits from coaching by an expert facilitator. However, in physical spaces it is not always easy to provide teams with evidence to support collaboration. Emerging technology provides a promising opportunity to make collocated collaboration visible by harnessing data about interactions and then mining and visualizing it. These collocated collaboration analytics can help researchers, designers, and users to understand the complexity of collaboration and to find ways they can support collaboration. This article introduces and motivates a set of principles for mining collocated collaboration data and draws attention to trade-offs that may need to be negotiated en route. We integrate Data Science principles and techniques with the advances in interactive surface devices and sensing technologies. We draw on a 7-year research program that has involved the analysis of six group situations in collocated settings with more than 500 users and a variety of surface technologies, tasks, grouping structures, and domains. The contribution of the article includes the key insights and themes that we have identified and summarized in a set of principles and dilemmas that can inform design of future collocated collaboration analytics innovations

The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

Digital Media Production to Support Literacy for Secondary Students with Diverse Learning Abilities

Producing digital media is a hands-on, inquiry-based mindful process that naturally embeds Universal Design for Learning (UDL) principles into literacy instruction, providing options for learning and assessment for a wide array of students with diverse learning abilities. Video production learning experiences acknowledge the cognitive talents of some students labeled “disabled.” For some, the discovery of personal abilities activated when learning through the production process may motivate deeper learning. Although challenges of access, quality of teacher preparation and assessment strategies represent significant challenges, digital media production learning experiences offer diverse learners a rich, socially interactive environment that models open communication and excitement for learning, and supports the scaffolding of comprehension skills for learning academic content

A review on data fusion in multimodal learning analytics and educational data mining

The new educational models such as smart learning environments use of digital and context-aware devices to facilitate the learning process. In this new educational scenario, a huge quantity of multimodal students' data from a variety of different sources can be captured, fused, and analyze. It offers to researchers and educators a unique opportunity of being able to discover new knowledge to better understand the learning process and to intervene if necessary. However, it is necessary to apply correctly data fusion approaches and techniques in order to combine various sources of multimodal learning analytics (MLA). These sources or modalities in MLA include audio, video, electrodermal activity data, eye-tracking, user logs, and click-stream data, but also learning artifacts and more natural human signals such as gestures, gaze, speech, or writing. This survey introduces data fusion in learning analytics (LA) and educational data mining (EDM) and how these data fusion techniques have been applied in smart learning. It shows the current state of the art by reviewing the main publications, the main type of fused educational data, and the data fusion approaches and techniques used in EDM/LA, as well as the main open problems, trends, and challenges in this specific research area

Design-activity-sequence: A case study and polyphonic analysis of learning in a digital design thinking workshop

In this case study, we report on the outcomes of a one-day workshop on design thinking attended by participants from the Computer-Supported Collaborative Learning conference in Philadelphia in 2017. We highlight the interactions between the workshop design, structured as a design thinking process around the design of a digital environment for design thinking, and the diverse backgrounds and interests of its participants. Data from in-workshop reflections and post-workshop interviews were analyzed using a novel set of analytical approaches, a combination the facilitators made by possible by welcoming participants as coresearchers

Temporal pathways to learning: how learning emerges in an open-ended collaborative activity

The learning process depends on the nature of the learning environment, particularly in the case of open-ended learning environments, where the learning process is considered to be non-linear. In this paper, we report on the findings of employing a multimodal Hidden Markov Model (HMM) based methodology to investigate the temporal learning processes of two types of learners that have learning gains and a type that does not have learning gains in an open-ended collaborative learning activity. Considering log data, speech behavior, affective states and gaze patterns, we find that all learners start from a similar state of non-productivity, but once out of it they are unlikely to fall back into that state, especially in the case of the learners that have learning gains. Those who have learning gains shift between two problem solving strategies, each characterized by both exploratory and reflective actions, as well as demonstrate speech and gaze patterns associated with these strategies, that differ from those who don't have learning gains. Further, the teams that have learning gains also differ between themselves in the manner in which they employ the problem solving strategies over the interaction, as well as in the manner they express negative emotions while exhibiting a particular strategy. These outcomes contribute to understanding the multiple pathways of learning in an open-ended collaborative learning environment, and provide actionable insights for designing effective interventions

Enhancing Free-text Interactions in a Communication Skills Learning Environment

Learning environments frequently use gamification to enhance user interactions.Virtual characters with whom players engage in simulated conversations often employ prescripted dialogues; however, free user inputs enable deeper immersion and higher-order cognition. In our learning environment, experts developed a scripted scenario as a sequence of potential actions, and we explore possibilities for enhancing interactions by enabling users to type free inputs that are matched to the pre-scripted statements using Natural Language Processing techniques. In this paper, we introduce a clustering mechanism that provides recommendations for fine-tuning the pre-scripted answers in order to better match user inputs