Multimodal Visual Sensing: Automated Estimation of Engagement

Viele moderne Anwendungen der künstlichen Intelligenz beinhalten bis zu einem gewissen Grad ein Verständnis der menschlichen Aufmerksamkeit, Aktivität, Absicht und Kompetenz aus multimodalen visuellen Daten. Nonverbale Verhaltenshinweise, die mit Hilfe von Computer Vision und Methoden des maschinellen Lernens erkannt werden, enthalten wertvolle Informationen zum Verständnis menschlicher Verhaltensweisen, einschließlich Aufmerksamkeit und Engagement. Der Einsatz solcher automatisierten Methoden im Bildungsbereich birgt ein enormes Potenzial. Zu den nützlichen Anwendungen gehören Analysen im Klassenzimmer zur Messung der Unterrichtsqualität und die Entwicklung von Interventionen zur Verbesserung des Unterrichts auf der Grundlage dieser Analysen sowie die Analyse von Präsentationen, um Studenten zu helfen, ihre Botschaften überzeugend und effektiv zu vermitteln. Diese Dissertation stellt ein allgemeines Framework vor, das auf multimodaler visueller Erfassung basiert, um Engagement und verwandte Aufgaben anhand visueller Modalitäten zu analysieren. Während sich der Großteil der Engagement-Literatur im Bereich des affektiven und sozialen Computings auf computerbasiertes Lernen und auf Lernspiele konzentriert, untersuchen wir die automatisierte Engagement-Schätzung im Klassenzimmer unter Verwendung verschiedener nonverbaler Verhaltenshinweise und entwickeln Methoden zur Extraktion von Aufmerksamkeits- und emotionalen Merkmalen. Darüber hinaus validieren wir die Effizienz der vorgeschlagenen Ansätze an realen Daten, die aus videografierten Klassen an Universitäten und weiterführenden Schulen gesammelt wurden. Zusätzlich zu den Lernaktivitäten führen wir eine Verhaltensanalyse von Studenten durch, die kurze wissenschaftliche Präsentationen unter Verwendung von multimodalen Hinweisen, einschließlich Gesichts-, Körper- und Stimmmerkmalen, halten. Neben dem Engagement und der Präsentationskompetenz nähern wir uns dem Verständnis des menschlichen Verhaltens aus einer breiteren Perspektive, indem wir die Analyse der gemeinsamen Aufmerksamkeit in einer Gruppe von Menschen, die Wahrnehmung von Lehrern mit Hilfe von egozentrischer Kameraperspektive und mobilen Eyetrackern sowie die automatisierte Anonymisierung von audiovisuellen Daten in Studien im Klassenzimmer untersuchen. Educational Analytics bieten wertvolle Möglichkeiten zur Verbesserung von Lernen und Lehren. Die Arbeit in dieser Dissertation schlägt einen rechnerischen Rahmen zur Einschätzung des Engagements und der Präsentationskompetenz von Schülern vor, zusammen mit unterstützenden Computer-Vision-Problemen.Many modern applications of artificial intelligence involve, to some extent, an understanding of human attention, activity, intention, and competence from multimodal visual data. Nonverbal behavioral cues detected using computer vision and machine learning methods include valuable information for understanding human behaviors, including attention and engagement. The use of such automated methods in educational settings has a tremendous potential for good. Beneficial uses include classroom analytics to measure teaching quality and the development of interventions to improve teaching based on these analytics, as well as presentation analysis to help students deliver their messages persuasively and effectively. This dissertation presents a general framework based on multimodal visual sensing to analyze engagement and related tasks from visual modalities. While the majority of engagement literature in affective and social computing focuses on computer-based learning and educational games, we investigate automated engagement estimation in the classroom using different nonverbal behavioral cues and developed methods to extract attentional and emotional features. Furthermore, we validate the efficiency of proposed approaches on real-world data collected from videotaped classes at university and secondary school. In addition to learning activities, we perform behavior analysis on students giving short scientific presentations using multimodal cues, including face, body, and voice features. Besides engagement and presentation competence, we approach human behavior understanding from a broader perspective by studying the analysis of joint attention in a group of people, teachers' perception using egocentric camera view and mobile eye trackers, and automated anonymization of audiovisual data in classroom studies. Educational analytics present valuable opportunities to improve learning and teaching. The work in this dissertation suggests a computational framework for estimating student engagement and presentation competence, together with supportive computer vision problems

D1.3 List of available solutions

This report has been submitted by Tempesta Media SL as deliverable D1.3 within the framework of H2020 project "SO-CLOSE: Enhancing Social Cohesion through Sharing the Cultural Heritage of Forced Migrations" Grant No. 870939.This report aims to conduct research on the specific topics and needs of the SO-CLOSE project, addressing the available solutions through a state-of-the-art digital tools analysis, applied in the cultural heritage and migration fields. More specifically the report's scope is:To define proper tools and proceedings for the interview needs -performing, recording, transcription, translation. To analyse potential content gathering tools for the co-creation workshops. To conduct a state-of-the-art sharing tools analysis, applied in the cultural heritage and migration fields, and propose a critically adjusted and innovative digital approach

On the Use of YouTube, Digital Games, Argument Maps, and Digital Feedback in Teaching Philosophy

We give an overview of the methodological possibilities of some important digital tools for teaching philosophy. Several didactically applicable methods have evolved in digital culture, including their implicit methodologies, theories about how these methods may be used. These methodologies are already applied by philosophers today and have their benefits and justifications in philosophy classes as well. They can help to solve known problems of philosophy education. We discuss problems of incomprehensibility and their possible solutions through digital explanations in pod- and videocasts such as YouTube; problems of interaction, motivation, and immersion that digital games and gamification may solve; problems of the complexity of philosophical content and digital concept- and argument-maps to deal with these; problems of implicitness and the possibility to make implicit things in philosophy class explicit through indirect feedback tools

Advancing video research methodology to capture the processes of social interaction and multimodality

In this reflective methodological paper we focus on affordances and challenges of video data. We compare and analyze two research settings that use the latest video technology to capture classroom interactions in mathematics education, namely, The Social Unit of Learning (SUL) project of the University of Melbourne and the MathTrack project of the University of Helsinki. While using these two settings as examples, we have structured our reflections around themes pertinent to video research in general, namely, research methods, data management, and research ethics. SUL and MathTrack share an understanding of mathematics learning as social multimodal practice, and provide possibilities for zooming into the situational micro interactions that construct collaborative problem-solving learning. Both settings provide rich data for in-depth analyses of peer interactions and learning processes. The settings share special needs for technical support and data management, as well as attention to ethical aspects from the perspective of the participants' security and discretion. SUL data are especially suitable for investigating interactions on a broad scope, addressing how multiple interactional processes intertwine. MathTrack, on the other hand, enables exploration of participants' visual attention in detail and its role in learning. Both settings could provide tools for teachers' professional development by showing them aspects of classroom interactions that would otherwise remain hidden.Peer reviewe

ProsocialLearn: D2.5 evaluation strategy and protocols

This document describes the evaluation strategy for the assessment of game effectiveness, market value impact and ethics procedure to drive detailed planning of technical validation, short and longitudinal studies and market viability tests

Are You Still With Me? Continuous Engagement Assessment From a Robot's Point of View

Continuously measuring the engagement of users with a robot in a Human-Robot Interaction (HRI) setting paves the way toward in-situ reinforcement learning, improve metrics of interaction quality, and can guide interaction design and behavior optimization. However, engagement is often considered very multi-faceted and difficult to capture in a workable and generic computational model that can serve as an overall measure of engagement. Building upon the intuitive ways humans successfully can assess situation for a degree of engagement when they see it, we propose a novel regression model (utilizing CNN and LSTM networks) enabling robots to compute a single scalar engagement during interactions with humans from standard video streams, obtained from the point of view of an interacting robot. The model is based on a long-term dataset from an autonomous tour guide robot deployed in a public museum, with continuous annotation of a numeric engagement assessment by three independent coders. We show that this model not only can predict engagement very well in our own application domain but show its successful transfer to an entirely different dataset (with different tasks, environment, camera, robot and people). The trained model and the software is available to the HRI community, at https://github.com/LCAS/engagement_detector, as a tool to measure engagement in a variety of settings