Understanding Human Functioning & Enhancing Human Potential through Computational Methods

Presented online on October 8, 2020 at 2:00 p.m.Sidney D’Mello is an Associate Professor in the Institute of Cognitive Science and Department of Computer Science at the University of Colorado Boulder. His work lies at the intersection of the computing, cognitive, affective, social, and learning sciences. D’Mello is interested in the dynamic interplay between cognition and emotion while individuals and groups engage in complex real-world activities.Runtime: 55:09 minutesIt is generally accepted that computational methods can complement traditional approaches to understanding human functioning, including thoughts, feelings, behaviors, and social interactions. I suggest that their utility extends beyond a mere complementary role. They serve a necessary role when data is too large for manual analysis, an opportunistic role by addressing questions that are beyond the purview of traditional methods, and a promissory role in facilitating change when fully-automated computational models are embedded in closed-loop intelligent systems. Multimodal computational approaches provide further benefits by affording analysis of disparate constructs emerging across multiple types of interactions in diverse contexts. To illustrate, I will discuss a research program that use linguistic, paralinguistic, behavioral, and physiological signals for the analysis of individual, small group, multi-party, and human-computer interactions in the lab and in the wild with the goals of understanding cognitive, noncognitive, and socio-affective-cognitive processes while improving human efficiency, engagement, and effectiveness. I will also discuss how these ideas align with our new NSF National AI Institute on Student-AI Teaming and how you can get involved in the research

Knowledge Elicitation Methods for Affect Modelling in Education

Research on the relationship between affect and cognition in Artificial Intelligence in Education (AIEd) brings an important dimension to our understanding of how learning occurs and how it can be facilitated. Emotions are crucial to learning, but their nature, the conditions under which they occur, and their exact impact on learning for different learners in diverse contexts still needs to be mapped out. The study of affect during learning can be challenging, because emotions are subjective, fleeting phenomena that are often difficult for learners to report accurately and for observers to perceive reliably. Context forms an integral part of learners’ affect and the study thereof. This review provides a synthesis of the current knowledge elicitation methods that are used to aid the study of learners’ affect and to inform the design of intelligent technologies for learning. Advantages and disadvantages of the specific methods are discussed along with their respective potential for enhancing research in this area, and issues related to the interpretation of data that emerges as the result of their use. References to related research are also provided together with illustrative examples of where the individual methods have been used in the past. Therefore, this review is intended as a resource for methodological decision making for those who want to study emotions and their antecedents in AIEd contexts, i.e. where the aim is to inform the design and implementation of an intelligent learning environment or to evaluate its use and educational efficacy

Automated Gaze-Based Mind Wandering Detection during Computerized Learning in Classrooms

We investigate the use of commercial off-the-shelf (COTS) eye-trackers to automatically detect mind wandering—a phenomenon involving a shift in attention from task-related to task-unrelated thoughts—during computerized learning. Study 1 (N = 135 high-school students) tested the feasibility of COTS eye tracking while students learn biology with an intelligent tutoring system called GuruTutor in their classroom. We could successfully track eye gaze in 75% (both eyes tracked) and 95% (one eye tracked) of the cases for 85% of the sessions where gaze was successfully recorded. In Study 2, we used this data to build automated student-independent detectors of mind wandering, obtaining accuracies (mind wandering F1 = 0.59) substantially better than chance (F1 = 0.24). Study 3 investigated context-generalizability of mind wandering detectors, finding that models trained on data collected in a controlled laboratory more successfully generalized to the classroom than the reverse. Study 4 investigated gaze- and video- based mind wandering detection, finding that gaze-based detection was superior and multimodal detection yielded an improvement in limited circumstances. We tested live mind wandering detection on a new sample of 39 students in Study 5 and found that detection accuracy (mind wandering F1 = 0.40) was considerably above chance (F1 = 0.24), albeit lower than offline detection accuracy from Study 1 (F1 = 0.59), a finding attributable to handling of missing data. We discuss our next steps towards developing gaze-based attention-aware learning technologies to increase engagement and learning by combating mind wandering in classroom contexts

Instructor presence effect: Liking does not always lead to learning

The final publication is available at Elsevier via https://dx.doi.org/10.1016/j.compedu.2018.03.011 © 2018. This manuscript version is made available under the CC-BY-NC-ND 4.0 license https://creativecommons.org/licenses/by-nc-nd/4.0/Online education provides the opportunity to present lecture material to students in different formats or modalities, however there is debate about which lecture formats are best. Here, we conducted four experiments with 19–68 year old online participants to address the question of whether visuals of the instructor in online video lectures benefit learning. In Experiments 1 (N = 168) and 2 (N = 206) participants were presented with a lecture in one of three modalities (audio, audio with text, or audio with visuals of the instructor). Participants reported on their attentiveness – mind wandering (MW) – throughout the lecture and then completed a comprehension test. We found no evidence of an advantage for video lectures with visuals of the instructor in terms of a reduction in MW or increase in comprehension. In fact, we found evidence of a comprehension cost, suggesting that visuals of instructors in video lectures may act as a distractor. In Experiments 3 (N = 88) and 4 (N = 109) we explored learners' subjective evaluations of lecture formats across 4 different lecture formats (audio, text, audio + text, audio + instructor, audio + text + instructor). The results revealed learners not only find online lectures with visuals of the instructor more enjoyable and interesting, they believe this format most facilitates their learning. Taken together, these results suggest visuals of the instructor potentially impairs comprehension, but learners prefer and believe they learn most effectively with this format. We refer to as the Instructor Presence Effect and discuss implications for multimedia learning and instructional design.Social Sciences and Humanities Research Council of Canada SSHRC) Insight Discovery Grant (70104)Canada Research Chairs program (056562)Early Researcher Award from the Province of Ontario (058402)Natural Sciences and Engineering Research Council of Canada (NSERC) Discovery Grant (RGPIN-2014-06459

Toward more effective and equitable learning: Identifying barriers and solutions for the future of online education

The increased reliance on online education and educational technologies more generally has laid bare the need to more deeply consider how researchers, designers, and educators can improve the quality of technology-mediated learning. To address this need, more than two dozen experts from a variety of fields came together to discuss the challenges that educational technology must address in the immediate future. These experts were tasked with identifying barriers to and potential solutions for delivering high-quality and equitable online and remote education. This article examines the themes and topics that emerged from these discussions and proposes a Collaborative Framework for Accelerating Online Education. This framework highlights the need for rapid experimentation within larger design cycles as well as the coordination and cooperation of multiple stakeholders across all phases of research and development. The themes, topics, and framework that emerged from this work serve as a call to action for innovative approaches to developing and studying online education. &nbsp;</p

Assessing Facial Expression Alignment in Couples Using CRQA

Data and code for NSF funded project (EHR #1745442) on interpersonal coordination and coregulation