Training Effects of Adaptive Emotive Responses From Animated Agents in Simulated Environments

Humans are distinct from machines in their capacity to emote, stimulate, and express emotions. Because emotions play such an important role in human interactions, human-like agents used in pedagogical roles for simulation-based training should properly reflect emotions. Currently, research concerning the development of this type of agent focuses on basic agent interface characteristics, as well as character building qualities. However, human-like agents should provide emotion-like qualities that are clearly expressed, properly synchronized, and that simulate complex, real-time interactions through adaptive emotion systems. The research conducted for this dissertation was a quantitative investigation using 3 (within) x 2 (between) x 3 (within) factorial design. A total of 56 paid participants consented to complete the study. Independent variables included emotion intensity (i.e., low, moderate, and high emotion), levels of expertise (novice participant versus experienced participant), and number of trials. Dependent measures included visual attention, emotional response towards the animated agents, simulation performance score, and learners\u27 perception of the pedagogical agent persona while participants interacted with a pain assessment and management simulation. While no relationships were indicated between the levels of emotion intensity portrayed by the animated agents and the participants\u27 visual attention, emotional response towards the animated agent, and simulation performance score, there were significant relationships between the level of expertise of the participant and the visual attention, emotional responses, and performance outcomes. The results indicated that nursing students had higher visual attention during their interaction with the animated agents. Additionally, nursing students expressed more neutral facial expression whereas experienced nurses expressed more emotional facial expressions towards the animated agents. The results of the simulation performance scores indicated that nursing students obtained higher performance scores in the pain assessment and management task than experienced nurses. Both groups of participants had a positive perception of the animated agents persona

Affective Computational Model to Extract Natural Affective States of Students with Asperger Syndrome (AS) in Computer-based Learning Environment

This study was inspired by looking at the central role of emotion in the learning process, its impact on students’ performance; as well as the lack of affective computing models to detect and infer affective-cognitive states in real time for students with and without Asperger Syndrome (AS). This model overcomes gaps in other models that were designed for people with autism, which needed the use of sensors or physiological instrumentations to collect data. The model uses a webcam to capture students’ affective-cognitive states of confidence, uncertainty, engagement, anxiety, and boredom. These states have a dominant effect on the learning process. The model was trained and tested on a natural-spontaneous affective dataset for students with and without AS, which was collected for this purpose. The dataset was collected in an uncontrolled environment and included variations in culture, ethnicity, gender, facial and hairstyle, head movement, talking, glasses, illumination changes and background variation. The model structure used deep learning (DL) techniques like convolutional neural network (CNN) and long short-term memory (LSTM). DL is the-state-of-art tool that used to reduce data dimensionality and capturing non-linear complex features from simpler representations. The affective model provide reliable results with accuracy 90.06%. This model is the first model to detected affective states for adult students with AS without physiological or wearable instruments. For the first time, the occlusions in this model, like hand over face or head were considered an important indicator for affective states like boredom, anxiety, and uncertainty. These occlusions have been ignored in most other affective models. The essential information channels in this model are facial expressions, head movement, and eye gaze. The model can serve as an aided-technology for tutors to monitor and detect the behaviors of all students at the same time and help in predicting negative affective states during learning process

Modeling Student Affective State Patterns during Self-Regulated Learning in Physics Playground

This dissertation research focuses on investigating the incidence of student self-regulated learning behavior, and examines patterns in student affective states that accompany such self-regulated behavior. This dissertation leverages prediction models of student affective states in the Physics Playground educational game platform to identify common patterns in student affective states during use of self-regulated learning behavior. In Study 1, prediction models of student affective states are developed in the context of the educational game environment Physics Playground, using affective state observations and computer log data that had already been collected as part of a larger project. The performances of student affective state prediction models generated using a combination of the computer log and observational data are then compared against those of similar prediction models generated using video data collected at the same time. In Study 2, I apply these affective state prediction models to generate predictions of student affective states on a broader set of data collected from students participants playing Physics Playground. In parallel, I define aggregated behavioral features that represent the self-observation and strategic planning components of self-regulated learning. Affective state predictions are then mapped to playground level attempts that contain these self-regulated learning behavioral features, and sequential pattern mining is applied to the affective state predictions to identify the most common patterns in student emotions. Findings from Study 1 demonstrate that both video data and interaction log data can be used to predict student affective states with significant accuracy. Since the video data is a direct measure of student emotions, it shows better performance across most affective states. However, the interaction log data can be collected natively by Physics Playground and is able to be generalized more easily to other learning environments. Findings from Study 2 suggest that self-regulatory behavior is closely associated with sustained periods of engaged concentration and .self-regulated learning behaviors are associated with transitions from negative affective states (confusion, frustration, and boredom) to the positive engaged concentration state. The results of this dissertation project demonstrate the power of measuring student affective states in real time and examining the temporal relationship to self-regulated learning behavior within an unstructured educational game platform. These results thus provide a building block for future research on the real-time assessment of student emotions and its relationship with self-regulated learning behaviors, particularly within online student-centered and self-directed learning contexts

A Novel Multimodal Approach for Studying the Dynamics of Curiosity in Small Group Learning

Curiosity is a vital metacognitive skill in educational contexts, leading to creativity, and a love of learning. And while many school systems increasingly undercut curiosity by teaching to the test, teachers are increasingly interested in how to evoke curiosity in their students to prepare them for a world in which lifelong learning and reskilling will be more and more important. One aspect of curiosity that has received little attention, however, is the role of peers in eliciting curiosity. We present what we believe to be the first theoretical framework that articulates an integrated socio-cognitive account of curiosity that ties observable behaviors in peers to underlying curiosity states. We make a bipartite distinction between individual and interpersonal functions that contribute to curiosity, and multimodal behaviors that fulfill these functions. We validate the proposed framework by leveraging a longitudinal latent variable modeling approach. Findings confirm a positive predictive relationship between the latent variables of individual and interpersonal functions and curiosity, with the interpersonal functions exercising a comparatively stronger influence. Prominent behavioral realizations of these functions are also discovered in a data-driven manner. We instantiate the proposed theoretical framework in a set of strategies and tactics that can be incorporated into learning technologies to indicate, evoke, and scaffold curiosity. This work is a step towards designing learning technologies that can recognize and evoke moment-by-moment curiosity during learning in social contexts and towards a more complete multimodal learning analytics. The underlying rationale is applicable more generally for developing computer support for other metacognitive and socio-emotional skills.Comment: arXiv admin note: text overlap with arXiv:1704.0748

The role of co-occurring emotions and personality traits in anger expression

The main aim of the current study was to examine the role of co-occurring emotions and their interactive effects with the Big Five personality traits in anger expression. Everyday anger expression ("anger-in" and "anger-out" behavior) was studied with the experience-sampling method in a group of 110 participants for 14 consecutive days on 7 random occasions per day. Our results showed that the simultaneously co-occurring emotions that buffer against anger expression are sadness, surprise, disgust, disappointment, and irritation for anger-in behavior, and fear, sadness and disappointment for anger-out reactions. While previous studies have shown that differentiating one's current affect into discrete emotion categories buffers against anger expression (Pond et al., 2012), our study further demonstrated the existence of specific interactive effects between the experience of momentary emotions and personality traits that lead to higher levels of either suppression or expression of anger behavior (or both). For example, the interaction between the trait Openness and co-occurring surprise, in predicting anger-in behavior, indicates that less open people hold their anger back more, and more open people use less anger-in behavior. Co-occurring disgust increases anger-out reactions in people low in Conscientiousness, but decreases anger-out reactions in people high in Conscientiousness. People high in Neuroticism are less likely to engage in anger-in behavior when experiencing disgust, surprise, or irritation alongside anger, but show more anger out in the case of co-occurring contempt. The results of the current study help to further clarify the interactions between the basic personality traits and the experience of momentary co-occurring emotions in determining anger behavior

Affective modelling and feedback in programming practice systems

指導教員：角　

Mitigating User Frustration through Adaptive Feedback based on Human-Automation Etiquette Strategies

The objective of this study is to investigate the effects of feedback and user frustration in human-computer interaction (HCI) and examine how to mitigate user frustration through feedback based on human-automation etiquette strategies. User frustration in HCI indicates a negative feeling that occurs when efforts to achieve a goal are impeded. User frustration impacts not only the communication with the computer itself, but also productivity, learning, and cognitive workload. Affect-aware systems have been studied to recognize user emotions and respond in different ways. Affect-aware systems need to be adaptive systems that change their behavior depending on users’ emotions. Adaptive systems have four categories of adaptations. Previous research has focused on primarily function allocation and to a lesser extent information content and task scheduling. However, the fourth approach, changing the interaction styles is the least explored because of the interplay of human factors considerations. Three interlinked studies were conducted to investigate the consequences of user frustration and explore mitigation techniques. Study 1 showed that delayed feedback from the system led to higher user frustration, anger, cognitive workload, and physiological arousal. In addition, delayed feedback decreased task performance and system usability in a human-robot interaction (HRI) context. Study 2 evaluated a possible approach of mitigating user frustration by applying human-human etiquette strategies in a tutoring context. The results of Study 2 showed that changing etiquette strategies led to changes in performance, motivation, confidence, and satisfaction. The most effective etiquette strategies changed when users were frustrated. Based on these results, an adaptive tutoring system prototype was developed and evaluated in Study 3. By utilizing a rule set derived from Study 2, the tutor was able to use different automation etiquette strategies to target and improve motivation, confidence, satisfaction, and performance using different strategies, under different levels of user frustration. This work establishes that changing the interaction style alone of a computer tutor can affect a user’s motivation, confidence, satisfaction, and performance. Furthermore, the beneficial effect of changing etiquette strategies is greater when users are frustrated. This work provides a basis for future work to develop affect-aware adaptive systems to mitigate user frustration

Towards Integration of Cognitive Models in Dialogue Management: Designing the Virtual Negotiation Coach Application

This paper presents an approach to flexible and adaptive dialogue management driven by cognitive modelling of human dialogue behaviour. Artificial intelligent agents, based on the ACT-R cognitive architecture, together with human actors are participating in a (meta)cognitive skills training within a negotiation scenario. The agent  employs instance-based learning to decide about its own actions and to reflect on the behaviour of the opponent. We show that task-related actions can be handled by a cognitive agent who is a plausible dialogue partner.  Separating task-related and dialogue control actions enables the application of sophisticated models along with a flexible architecture  in which  various alternative modelling methods can be combined. We evaluated the proposed approach with users assessing  the relative contribution of various factors to the overall usability of a dialogue system. Subjective perception of effectiveness, efficiency and satisfaction were correlated with various objective performance metrics, e.g. number of (in)appropriate system responses, recovery strategies, and interaction pace. It was observed that the dialogue system usability is determined most by the quality of agreements reached in terms of estimated Pareto optimality, by the user's negotiation strategies selected, and by the quality of system recognition, interpretation and responses. We compared human-human and human-agent performance with respect to the number and quality of agreements reached, estimated cooperativeness level, and frequency of accepted negative outcomes. Evaluation experiments showed promising, consistently positive results throughout the range of the relevant scales

A Novel Multimodal Approach for Studying the Dynamics of Curiosity in Small Group Learning

Curiosity is a vital metacognitive skill in educational contexts, leading to creativity, and a love of learning. And while many school systems increasingly undercut curiosity by teaching to the test, teachers are increasingly interested in how to evoke curiosity in their students to prepare them for a world in which lifelong learning and reskilling will be more and more important. One aspect of curiosity that has received little attention, however, is the role of peers in eliciting curiosity. We present what we believe to be the first theoretical framework that articulates an integrated socio-cognitive account of curiosity that ties observable behaviors in peers to underlying curiosity states. We make a bipartite distinction between individual and interpersonal functions that contribute to curiosity, and multimodal behaviors that fulfill these functions. We validate the proposed framework by leveraging a longitudinal latent variable modeling approach. Findings confirm a positive predictive relationship between the latent variables of individual and interpersonal functions and curiosity, with the interpersonal functions exercising a comparatively stronger influence. Prominent behavioral realizations of these functions are also discovered in a data-driven manner. We instantiate the proposed theoretical framework in a set of strategies and tactics that can be incorporated into learning technologies to indicate, evoke, and scaffold curiosity. This work is a step towards designing learning technologies that can recognize and evoke moment-by-moment curiosity during learning in social contexts and towards a more complete multimodal learning analytics. The underlying rationale is applicable more generally for developing computer support for other metacognitive and socio-emotional skills

Motivational and metacognitive feedback in an ITS: linking past states and experiences to current problems

Feedback is an important element in learning as it can provide learners with both information about progress as well as external motivational stimuli, providing them with an opportunity for reflection. Motivation and metacognition are strongly intertwined, with learners high in self-efficacy more likely to use a variety of self-regulatory learning strategies, as well as to persist longer on challenging tasks. Learning from past experience involves metacognitive processes as an act of reflecting upon one’s own experience and, coupled with existing knowledge, aids the acquisition and construction of further knowledge. The aim of the research was to improve the learner’s focus on the process and experience of problem solving while using an Intelligent Tutoring System (ITS), by addressing the primary question: what are the effects of including motivational and metacognitive feedback based on the learner’s past states and experiences? An existing ITS, SQL-Tutor, was used in a study with participants from first year undergraduate degrees studying a database module. The study used two versions of SQL-Tutor: the Control group used a base version providing domain feedback and the Study group used an extended version that also provided motivational and metacognitive feedback. Three sources of data collection were used: module summative assessments, ITS log files and a post-study questionnaire. The analysis included both pre-post comparisons and how the participants interacted with the system, for example their persistence in problem-solving and the degree to which they referred to past learning. Comparisons between groups showed some differing trends both in learning and behaviour in favour of the Study group, though these trends were not significantly different. The study findings showed promise for the use of motivational and metacognitive feedback based on the learners’ past states and experiences that could be used as a basis for future research work and refinement