Automatic Context-Driven Inference of Engagement in HMI: A Survey

An integral part of seamless human-human communication is engagement, the process by which two or more participants establish, maintain, and end their perceived connection. Therefore, to develop successful human-centered human-machine interaction applications, automatic engagement inference is one of the tasks required to achieve engaging interactions between humans and machines, and to make machines attuned to their users, hence enhancing user satisfaction and technology acceptance. Several factors contribute to engagement state inference, which include the interaction context and interactants' behaviours and identity. Indeed, engagement is a multi-faceted and multi-modal construct that requires high accuracy in the analysis and interpretation of contextual, verbal and non-verbal cues. Thus, the development of an automated and intelligent system that accomplishes this task has been proven to be challenging so far. This paper presents a comprehensive survey on previous work in engagement inference for human-machine interaction, entailing interdisciplinary definition, engagement components and factors, publicly available datasets, ground truth assessment, and most commonly used features and methods, serving as a guide for the development of future human-machine interaction interfaces with reliable context-aware engagement inference capability. An in-depth review across embodied and disembodied interaction modes, and an emphasis on the interaction context of which engagement perception modules are integrated sets apart the presented survey from existing surveys

Affect-driven Engagement Measurement from Videos

In education and intervention programs, person's engagement has been identified as a major factor in successful program completion. Automatic measurement of person's engagement provides useful information for instructors to meet program objectives and individualize program delivery. In this paper, we present a novel approach for video-based engagement measurement in virtual learning programs. We propose to use affect states, continuous values of valence and arousal extracted from consecutive video frames, along with a new latent affective feature vector and behavioral features for engagement measurement. Deep learning-based temporal, and traditional machine-learning-based non-temporal models are trained and validated on frame-level, and video-level features, respectively. In addition to the conventional centralized learning, we also implement the proposed method in a decentralized federated learning setting and study the effect of model personalization in engagement measurement. We evaluated the performance of the proposed method on the only two publicly available video engagement measurement datasets, DAiSEE and EmotiW, containing videos of students in online learning programs. Our experiments show a state-of-the-art engagement level classification accuracy of 63.3% and correctly classifying disengagement videos in the DAiSEE dataset and a regression mean squared error of 0.0673 on the EmotiW dataset. Our ablation study shows the effectiveness of incorporating affect states in engagement measurement. We interpret the findings from the experimental results based on psychology concepts in the field of engagement.Comment: 13 pages, 8 figures, 7 table

Automated Gaze-Based Mind Wandering Detection during Computerized Learning in Classrooms

We investigate the use of commercial off-the-shelf (COTS) eye-trackers to automatically detect mind wandering—a phenomenon involving a shift in attention from task-related to task-unrelated thoughts—during computerized learning. Study 1 (N = 135 high-school students) tested the feasibility of COTS eye tracking while students learn biology with an intelligent tutoring system called GuruTutor in their classroom. We could successfully track eye gaze in 75% (both eyes tracked) and 95% (one eye tracked) of the cases for 85% of the sessions where gaze was successfully recorded. In Study 2, we used this data to build automated student-independent detectors of mind wandering, obtaining accuracies (mind wandering F1 = 0.59) substantially better than chance (F1 = 0.24). Study 3 investigated context-generalizability of mind wandering detectors, finding that models trained on data collected in a controlled laboratory more successfully generalized to the classroom than the reverse. Study 4 investigated gaze- and video- based mind wandering detection, finding that gaze-based detection was superior and multimodal detection yielded an improvement in limited circumstances. We tested live mind wandering detection on a new sample of 39 students in Study 5 and found that detection accuracy (mind wandering F1 = 0.40) was considerably above chance (F1 = 0.24), albeit lower than offline detection accuracy from Study 1 (F1 = 0.59), a finding attributable to handling of missing data. We discuss our next steps towards developing gaze-based attention-aware learning technologies to increase engagement and learning by combating mind wandering in classroom contexts

Multimodal Data Analysis of Dyadic Interactions for an Automated Feedback System Supporting Parent Implementation of Pivotal Response Treatment

abstract: Parents fulfill a pivotal role in early childhood development of social and communication skills. In children with autism, the development of these skills can be delayed. Applied behavioral analysis (ABA) techniques have been created to aid in skill acquisition. Among these, pivotal response treatment (PRT) has been empirically shown to foster improvements. Research into PRT implementation has also shown that parents can be trained to be effective interventionists for their children. The current difficulty in PRT training is how to disseminate training to parents who need it, and how to support and motivate practitioners after training. Evaluation of the parents’ fidelity to implementation is often undertaken using video probes that depict the dyadic interaction occurring between the parent and the child during PRT sessions. These videos are time consuming for clinicians to process, and often result in only minimal feedback for the parents. Current trends in technology could be utilized to alleviate the manual cost of extracting data from the videos, affording greater opportunities for providing clinician created feedback as well as automated assessments. The naturalistic context of the video probes along with the dependence on ubiquitous recording devices creates a difficult scenario for classification tasks. The domain of the PRT video probes can be expected to have high levels of both aleatory and epistemic uncertainty. Addressing these challenges requires examination of the multimodal data along with implementation and evaluation of classification algorithms. This is explored through the use of a new dataset of PRT videos. The relationship between the parent and the clinician is important. The clinician can provide support and help build self-efficacy in addition to providing knowledge and modeling of treatment procedures. Facilitating this relationship along with automated feedback not only provides the opportunity to present expert feedback to the parent, but also allows the clinician to aid in personalizing the classification models. By utilizing a human-in-the-loop framework, clinicians can aid in addressing the uncertainty in the classification models by providing additional labeled samples. This will allow the system to improve classification and provides a person-centered approach to extracting multimodal data from PRT video probes.Dissertation/ThesisDoctoral Dissertation Computer Science 201

Tracking in the wild: exploring the everyday use of physical activity trackers

As the rates of chronical diseases, such as obesity, cardiovascular disease and diabetes continue to increase, the development of tools that support people in achieving healthier habits is becoming ever more important. Personal tracking systems, such as activity trackers, have emerged as a promising class of tools to support people in managing their everyday health. However, for this promise to be fulfilled, these systems need to be well designed, not only in terms of how they implement specific behavior change techniques, but also in how they integrate into people’s daily lives and address their daily needs. My dissertations provides evidence that accounting for people’s daily practices and needs can help to design activity tracking systems that help people get more value from their tracking practices. To understand how people derive value from their activity tracking practices, I have conducted two inquiries into people’s daily uses of activity tracking systems. In a fist attempt, I led a 10-month study of the adoption of Habito, our own activity tracking mobile app. Habito logged not only users’ physical activity, but also their interactions with the app. This data was used to acquire an estimate of the adoption rate of Habito, and understanding of how adoption is affected by users’ ‘readiness’, i.e., their attitude towards behavior change. In a follow-up study, I turned to the use of video methods and direct, in-situ observations of users’ interactions to understand what motivates people to engage with these tools in their everyday life, and how the surrounding environment shapes their use. These studies revealed some of the complexities of tracking, while extending some of the underlying ideas of behavior change. Among key results: (1) people’s use of activity trackers was found to be predominantly impulsive, where they simultaneously reflect, learn and change their behaviors as they collect data; (2) people’s use of trackers is deeply entangled with their daily routines and practices, and; (3) people use of trackers often is not in line with the traditional vision of these tools as mediators of change – trackers are also commonly used to simply learn about behaviors and engage in moments of self-discovery. Examining how to design activity tracking interfaces that best support people’s different needs , my dissertation further describes an inquiry into the design space of behavioral feedback interfaces. Through a iterative process of synthesis and analysis of research on activity tracking, I devise six design qualities for creating feedback that supports people in their interactions with physical activity data. Through the development and field deployment of four concepts in a field study, I show the potential of these displays for highlighting opportunities for action and learning.À medida que a prevalência de doenças crónicas como a obesidade, doenças cardiovasculares e diabetes continua a aumentar, o desenvolvimento de ferramentas que suportam pessoas a atingir mudanças de comportamento tem-se tornado essencial. Ferramentas de monitorização de comportamentos, tais como monitores de atividade física, têm surgido com a promessa de encorajar um dia a dia mais saudável. Contudo, para que essa promessa seja cumprida, torna-se essencial que estas ferramentas sejam bem concebidas, não só na forma como implementam determinadas estratégias de mudança de comportamento, mas também na forma como são integradas no dia-a-dia das pessoas. A minha dissertação demonstra a importância de considerar as necessidades e práticas diárias dos utilizadores destas ferramentas, de forma a ajudá-las a tirar melhor proveito da sua monitorização de atividade física. De modo a entender como é que os utilizadores destas ferramentas derivam valor das suas práticas de monitorização, a minha dissertação começa por explorar as práticas diárias associadas ao uso de monitores de atividade física. A minha dissertação contribui com duas investigações ao uso diário destas ferramentas. Primeiro, é apresentada uma investigação da adoção de Habito, uma aplicação para monitorização de atividade física. Habito não só registou as instâncias de atividade física dos seus utilizadores, mas também as suas interações com a própria aplicação. Estes dados foram utilizados para adquirir uma taxa de adopção de Habito e entender como é que essa adopção é afetada pela “prontidão” dos utilizadores, i.e., a sua atitude em relação à mudança de comportamento. Num segundo estudo, recorrendo a métodos de vídeo e observações diretas e in-situ da utilização de monitores de atividade física, explorei as motivações associadas ao uso diário destas ferramentas. Estes estudos expandiram algumas das ideias subjacentes ao uso das ferramentas para mudanças de comportamento. Entre resultados principais: (1) o uso de monitores de atividade física é predominantemente impulsivo, onde pessoas refletem, aprendem e alteram os seus comportamentos à medida que recolhem dados sobe estes mesmos comportamentos; (2) o uso de monitores de atividade física está profundamente interligado com as rotinas e práticas dos seus utilizadores, e; (3) o uso de monitores de atividade física nem sempre está ligado a mudanças de comportamento – estas ferramentas também são utilizadas para divertimento e aprendizagem. A minha dissertação contribui ainda com uma exploração do design de interfaces para a monitorização de atividade física. Através de um processo iterativo de síntese e análise de literatura, seis qualidades para a criação de interfaces são derivadas. Através de um estudo de campo, a minha dissertação demonstro o potencial dessas interfaces para ajudar pessoas a aprender e gerir a sua saúde diária

Approaches, applications, and challenges in physiological emotion recognition — a tutorial overview

An automatic emotion recognition system can serve as a fundamental framework for various applications in daily life from monitoring emotional well-being to improving the quality of life through better emotion regulation. Understanding the process of emotion manifestation becomes crucial for building emotion recognition systems. An emotional experience results in changes not only in interpersonal behavior but also in physiological responses. Physiological signals are one of the most reliable means for recognizing emotions since individuals cannot consciously manipulate them for a long duration. These signals can be captured by medical-grade wearable devices, as well as commercial smart watches and smart bands. With the shift in research direction from laboratory to unrestricted daily life, commercial devices have been employed ubiquitously. However, this shift has introduced several challenges, such as low data quality, dependency on subjective self-reports, unlimited movement-related changes, and artifacts in physiological signals. This tutorial provides an overview of practical aspects of emotion recognition, such as experiment design, properties of different physiological modalities, existing datasets, suitable machine learning algorithms for physiological data, and several applications. It aims to provide the necessary psychological and physiological backgrounds through various emotion theories and the physiological manifestation of emotions, thereby laying a foundation for emotion recognition. Finally, the tutorial discusses open research directions and possible solutions

Biometric features modeling to measure students engagement.

The ability to measure students’ engagement in an educational setting may improve student retention and academic success, revealing which students are disinterested, or which segments of a lesson are causing difficulties. This ability will facilitate timely intervention in both the learning and the teaching process in a variety of classroom settings. In this dissertation, an automatic students engagement measure is proposed through investigating three main engagement components of the engagement: the behavioural engagement, the emotional engagement and the cognitive engagement. The main goal of the proposed technology is to provide the instructors with a tool that could help them estimating both the average class engagement level and the individuals engagement levels while they give the lecture in real-time. Such system could help the instructors to take actions to improve students\u27 engagement. Also, it can be used by the instructor to tailor the presentation of material in class, identify course material that engages and disengages with students, and identify students who are engaged or disengaged and at risk of failure. A biometric sensor network (BSN) is designed to capture data consist of individuals facial capture cameras, wall-mounted cameras and high performance computing machine to capture students head pose, eye gaze, body pose, body movements, and facial expressions. These low level features will be used to train a machine-learning model to estimate the behavioural and emotional engagements in either e-learning or in-class environment. A set of experiments is conducted to compare the proposed technology with the state-of-the-art frameworks in terms of performance. The proposed framework shows better accuracy in estimating both behavioral and emotional engagement. Also, it offers superior flexibility to work in any educational environment. Further, this approach allows quantitative comparison of teaching methods, such as lecture, flipped classrooms, classroom response systems, etc. such that an objective metric can be used for teaching evaluation with immediate closed-loop feedback to the instructor