Affective games:a multimodal classification system

Affective gaming is a relatively new field of research that exploits human emotions to influence gameplay for an enhanced player experience. Changes in player’s psychology reflect on their behaviour and physiology, hence recognition of such variation is a core element in affective games. Complementary sources of affect offer more reliable recognition, especially in contexts where one modality is partial or unavailable. As a multimodal recognition system, affect-aware games are subject to the practical difficulties met by traditional trained classifiers. In addition, inherited game-related challenges in terms of data collection and performance arise while attempting to sustain an acceptable level of immersion. Most existing scenarios employ sensors that offer limited freedom of movement resulting in less realistic experiences. Recent advances now offer technology that allows players to communicate more freely and naturally with the game, and furthermore, control it without the use of input devices. However, the affective game industry is still in its infancy and definitely needs to catch up with the current life-like level of adaptation provided by graphics and animation

Toward emotional interactive videogames for children with autism spectrum disorder

Technology and videogames have been proven as motivating tools for working attention and complex communication skills, especially in children with autism spectrum disorder (ASD). In this work, we present two experiences that used interactive games for promoting communication and attention. The first game considers emotions in order to measure children’s attention, concentration and satisfaction, while the second uses tangible tabletops for fostering cognitive planning. The analysis of the results obtained allows to propose a new study integrating both, in which the tangible interactive game is complemented with the emotional trainer in a way that allows identifying and classifying children’s emotion with ASD when they collaborate to solve cognitively significant and contextualized challenges. The first application proposed is an emotional trainer application in which the child can work out the seven basic emotions (happiness, sadness, fear, disgust, anger, surprise and neutral). Further, a serious videogame is proposed: a 3D maze where the emotions can be captured. The second case study was carried out in a Special Education Center, where a set of activities for working cognitive planning was proposed. In this case, a tangible interactive tabletop was used to analyze, in students with ASD, how the communication processes with these interfaces affect to the attention, memory, successive and simultaneous processing that compose cognitive planning from the PASS model. The results of the first study, suggest that the autistic children did not act with previous planning, but they used their perception to adjust their actions a posteriori (that explains the higher number of collisions). On the second case study, the successive processing was not explored. The inclusion of the mazes of case study 1 to a semantic rich scenario could allow us to measure the prior planning and the emotions involved in the maze game. The new physiological sensors will also help to validate the emotions felt by the children. The first study has as objective the capability to imitate emotions and resolve a maze without semantic context. The second study organized all the actions from a semantic context close to users. The attention results presented by the second study are coherent with the first study and complement it showing that attention can be receptive or selective. In the first study case, the receptive attention was the focus of analysis. In the second case, both contributed to explain and understand how it can be developed from a videogame

Evaluating Mental Stress Among College Students Using Heart Rate and Hand Acceleration Data Collected from Wearable Sensors

Stress is various mental health disorders including depression and anxiety among college students. Early stress diagnosis and intervention may lower the risk of developing mental illnesses. We examined a machine learning-based method for identification of stress using data collected in a naturalistic study utilizing self-reported stress as ground truth as well as physiological data such as heart rate and hand acceleration. The study involved 54 college students from a large campus who used wearable wrist-worn sensors and a mobile health (mHealth) application continuously for 40 days. The app gathered physiological data including heart rate and hand acceleration at one hertz frequency. The application also enabled users to self-report stress by tapping on the watch face, resulting in a time-stamped record of the self-reported stress. We created, evaluated, and analyzed machine learning algorithms for identifying stress episodes among college students using heart rate and accelerometer data. The XGBoost method was the most reliable model with an AUC of 0.64 and an accuracy of 84.5%. The standard deviation of hand acceleration, standard deviation of heart rate, and the minimum heart rate were the most important features for stress detection. This evidence may support the efficacy of identifying patterns in physiological reaction to stress using smartwatch sensors and may inform the design of future tools for real-time detection of stress

Virtual reality exposure therapy for social phobia

This thesis presents researches and experiments performed in collaboration with a psychiatrist in order to validate and improve the use of virtual reality in social phobia psychotherapy. Cognitive and behavioral therapies are strongly based on the exposure to anxiety provoking stimuli. Virtual reality seems to be appropriate for such exposures as it allows for on-demand reproduction of reality. The idea has been validated for the treatment of various phobias but is more delicate in the case of social phobia; whereas the sense of presence provoked by the immersion in a virtual environment supports the emergence of fears linked to a location, we had to verify that we can reproduce social phobia related anxiety-provoking stimuli by simulating virtual humans. Therefore, and in order to provide therapists with an efficient virtual reality system dedicated to the exposure to social situations, we have developed software solutions supporting different immersion setups and enabling realistic simulations of inhabited virtual environments. We have experimented with public speaking scenarios within a preliminary study, three clinical case studies and a validation study on 200 subjects. We have been able to confirm that our virtual reality platform fulfilled therapeutic exposure requirements for social phobia. Moreover, we have been able to show that virtual reality exposure has additional advantages such as the possibility to improve clinical assessment with embedded monitoring tools. Our experiments with physiological measurements and eye tracking technology during immersion leaded to the validation of systems for objective and reliable assessment of patients' safety behaviors. The observation of such phobic reactions has confirmed the simulation impact and may provide therapists with enhanced pathological progression monitoring. During our experiments, we have also been able to observe that subjects' reactions during immersion were so much influenced by their sensitivity to fearful stimuli that their cognitive reactions were 'overloaded' by the arousal of anxiety and emotions. This has allowed us to consider that the sense of presence was more importantly related to the subjective impact of the content than to the technological process

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective

Modélisation des émotions de l’apprenant et interventions implicites pour les systèmes tutoriels intelligents

La modélisation de l’expérience de l’utilisateur dans les Interactions Homme-Machine est un enjeu important pour la conception et le développement des systèmes adaptatifs intelligents. Dans ce contexte, une attention particulière est portée sur les réactions émotionnelles de l’utilisateur, car elles ont une influence capitale sur ses aptitudes cognitives, comme la perception et la prise de décision. La modélisation des émotions est particulièrement pertinente pour les Systèmes Tutoriels Émotionnellement Intelligents (STEI). Ces systèmes cherchent à identifier les émotions de l’apprenant lors des sessions d’apprentissage, et à optimiser son expérience d’interaction en recourant à diverses stratégies d’interventions. Cette thèse vise à améliorer les méthodes de modélisation des émotions et les stratégies émotionnelles utilisées actuellement par les STEI pour agir sur les émotions de l’apprenant. Plus précisément, notre premier objectif a été de proposer une nouvelle méthode pour détecter l’état émotionnel de l’apprenant, en utilisant différentes sources d’informations qui permettent de mesurer les émotions de façon précise, tout en tenant compte des variables individuelles qui peuvent avoir un impact sur la manifestation des émotions. Pour ce faire, nous avons développé une approche multimodale combinant plusieurs mesures physiologiques (activité cérébrale, réactions galvaniques et rythme cardiaque) avec des variables individuelles, pour détecter une émotion très fréquemment observée lors des sessions d’apprentissage, à savoir l’incertitude. Dans un premier lieu, nous avons identifié les indicateurs physiologiques clés qui sont associés à cet état, ainsi que les caractéristiques individuelles qui contribuent à sa manifestation. Puis, nous avons développé des modèles prédictifs permettant de détecter automatiquement cet état à partir des différentes variables analysées, à travers l’entrainement d’algorithmes d’apprentissage machine. Notre deuxième objectif a été de proposer une approche unifiée pour reconnaître simultanément une combinaison de plusieurs émotions, et évaluer explicitement l’impact de ces émotions sur l’expérience d’interaction de l’apprenant. Pour cela, nous avons développé une plateforme hiérarchique, probabiliste et dynamique permettant de suivre les changements émotionnels de l'apprenant au fil du temps, et d’inférer automatiquement la tendance générale qui caractérise son expérience d’interaction à savoir : l’immersion, le blocage ou le décrochage. L’immersion correspond à une expérience optimale : un état dans lequel l'apprenant est complètement concentré et impliqué dans l’activité d’apprentissage. L’état de blocage correspond à une tendance d’interaction non optimale où l'apprenant a de la difficulté à se concentrer. Finalement, le décrochage correspond à un état extrêmement défavorable où l’apprenant n’est plus du tout impliqué dans l’activité d’apprentissage. La plateforme proposée intègre trois modalités de variables diagnostiques permettant d’évaluer l’expérience de l’apprenant à savoir : des variables physiologiques, des variables comportementales, et des mesures de performance, en combinaison avec des variables prédictives qui représentent le contexte courant de l’interaction et les caractéristiques personnelles de l'apprenant. Une étude a été réalisée pour valider notre approche à travers un protocole expérimental permettant de provoquer délibérément les trois tendances ciblées durant l’interaction des apprenants avec différents environnements d’apprentissage. Enfin, notre troisième objectif a été de proposer de nouvelles stratégies pour influencer positivement l’état émotionnel de l’apprenant, sans interrompre la dynamique de la session d’apprentissage. Nous avons à cette fin introduit le concept de stratégies émotionnelles implicites : une nouvelle approche pour agir subtilement sur les émotions de l’apprenant, dans le but d’améliorer son expérience d’apprentissage. Ces stratégies utilisent la perception subliminale, et plus précisément une technique connue sous le nom d’amorçage affectif. Cette technique permet de solliciter inconsciemment les émotions de l’apprenant, à travers la projection d’amorces comportant certaines connotations affectives. Nous avons mis en œuvre une stratégie émotionnelle implicite utilisant une forme particulière d’amorçage affectif à savoir : le conditionnement évaluatif, qui est destiné à améliorer de façon inconsciente l’estime de soi. Une étude expérimentale a été réalisée afin d’évaluer l’impact de cette stratégie sur les réactions émotionnelles et les performances des apprenants.Modeling the user’s experience within Human-Computer Interaction is an important challenge for the design and development of intelligent adaptive systems. In this context, a particular attention is given to the user’s emotional reactions, as they decisively influence his cognitive abilities, such as perception and decision-making. Emotion modeling is particularly relevant for Emotionally Intelligent Tutoring Systems (EITS). These systems seek to identify the learner’s emotions during tutoring sessions, and to optimize his interaction experience using a variety of intervention strategies. This thesis aims to improve current methods on emotion modeling, as well as the emotional strategies that are presently used within EITS to influence the learner’s emotions. More precisely, our first objective was to propose a new method to recognize the learner’s emotional state, using different sources of information that allow to measure emotions accurately, whilst taking account of individual characteristics that can have an impact on the manifestation of emotions. To that end, we have developed a multimodal approach combining several physiological measures (brain activity, galvanic responses and heart rate) with individual variables, to detect a specific emotion, which is frequently observed within computer tutoring, namely : uncertainty. First, we have identified the key physiological indicators that are associated to this state, and the individual characteristics that contribute to its manifestation. Then, we have developed predictive models to automatically detect this state from the analyzed variables, trough machine learning algorithm training. Our second objective was to propose a unified approach to simultaneously recognize a combination of several emotions, and to explicitly evaluate the impact of these emotions on the learner’s interaction experience. For this purpose, we have developed a hierarchical, probabilistic and dynamic framework, which allows one to track the learner’s emotional changes over time, and to automatically infer the trend that characterizes his interaction experience namely : flow, stuck or off-task. Flow is an optimal experience : a state in which the learner is completely focused and involved within the learning activity. The state of stuck is a non-optimal trend of the interaction where the learner has difficulty to maintain focused attention. Finally, the off-task behavior is an extremely unfavorable state where the learner is not involved anymore within the learning session. The proposed framework integrates three-modality diagnostic variables that sense the learner’s experience including : physiology, behavior and performance, in conjunction with predictive variables that represent the current context of the interaction and the learner’s personal characteristics. A human-subject study was conducted to validate our approach through an experimental protocol designed to deliberately elicit the three targeted trends during the learners’ interaction with different learning environments. Finally, our third objective was to propose new strategies to positively influence the learner’s emotional state, without interrupting the dynamics of the learning session. To this end, we have introduced the concept of implicit emotional strategies : a novel approach to subtly impact the learner’s emotions, in order to improve his learning experience. These strategies use the subliminal perception, and more precisely a technique known as affective priming. This technique aims to unconsciously solicit the learner’s emotions, through the projection of primes charged with specific affective connotations. We have implemented an implicit emotional strategy using a particular form of affective priming namely : the evaluative conditioning, which is designed to unconsciously enhance self-esteem. An experimental study was conducted in order to evaluate the impact of this strategy on the learners’ emotional reactions and performance

Design and Development of a Real-Time Bio-Sensing System Assessing Student Mental Workload and Engagement

Ο εντοπισμός του επακριβούς επιπέδου προσήλωσης και εμπλοκής των μαθητών με το περιεχόμενο διδασκαλίας στην τάξη είναι ένας από τους πιο μεγαλεπήβολους στόχους των ερευνητών της εκπαιδευτικής και επιστημονικής κοινότητας. (Lang, 1995, Grossberg, 1987). Σχετικές διεπιστημονικές ερευνητικές προσπάθειες προσαύξησης ενδιαφέροντος και εντοπισμού της αποτελεσματικότητας των διδακτικών πρακτικών βασίζονται σε τυπικές μελέτες από τον χώρο της ψυχολογίας, της παιδαγωγικής, της παιδοψυχολογίας και της ψυχοφυσιολογίας. Νέες τεχνολογίες έχουν εισάγει διαγνωστικές συσκευές δανεισμένες από τον χώρο της ιατρικής με σκοπό να εκμεταλλευτούν τις δυνατότητες μετρήσεων βιολογικών σημάτων τα οποία αποτελούν επιβεβαιωμένες εκφράσεις ψυχοφυσιολογικών καταστάσεων οι οποίες μπορούν να μεταφραστούν σε εκδηλώσεις διέγερσης και διάθεσης. Οι ιατρικές συσκευές απαιτούν εργαστηριακό περιβάλλον λόγω των αναγκών χρήσης ηλεκτροδίων, κινητικών περιορισμών, συγχρονισμού και ομοιομορφίας των στοιχείων που προκύπτουν και γι’ αυτό τον λόγο δεν μπόρεσαν ποτέ να αποδόσουν μια προσιτή λύση εφαρμόσιμη ευρύτερα σε εκπαιδευτικό περιβάλλον. Στην παρούσα μελέτη, αναλύονται οι επιδόσεις μιας ειδικά κατασκευασμένης ηλεκτρονικής συσκευής, σχεδιασμένης ώστε να εξεταστούν οι δυνατότητες να εξαχθούν δείκτες ψυχοσωματικών εκφράσεων του χρήστη, με την δυνατότητα να χρησιμοποιείται εύχρηστα στην τάξη χωρίς ηλεκτρόδια και επηρεασμούς από προσαρτήσεις. Το ολοκληρωμένο σύστημα μέτρησης και αποτύπωσης συμπερασμάτων είναι βασισμένο σε μοντελοποίηση συμπεριφορών αλλαγής του καρδιακού παλμού και της ειδικής διηλεκτρικής αγωγιμότητας του δέρματος σε πραγματικό χρόνο. Η συσκευή χρησιμοποιεί οπτικούς και διηλεκτρικούς αισθητήρες επαφής και έχει μελετηθεί σε αντιπαραβολή με διαβαθμισμένα περιβάλλοντα προκλητών καταστάσεων νοητικής φόρτισης. Σειρές πειραματικών διαδικασιών εφαρμοσμένες σε διαβαθμισμένα σενάρια πρόκλησης ψυχοσωματικών διεγέρσεων έχουν ολοκληρωθεί για επικύρωση, μελέτη επιδόσεων και λειτουργία του συστήματος ακόμη και σε σύγκριση με εμπορικό προϊόν. Πειραματικά αποτελέσματα δείχνουν αξιόλογους συσχετισμούς του μοντέλου και των επιδόσεων του συστήματος με τις αναμενόμενες αποκρίσεις με ενθαρρυντικά ποσοστά ακρίβειας.Facing the challenge of improving adaptive interaction in educational technologies scientists and educators have turned their focal point in diverse areas ranging from educational, teaching and behavioural psychology to cognitive, affective and perceptual neuroscience. The introduction of digital technologies and interactive media tools in education has shown improved learning efficiency, much higher memory activation and assimilation than verbal teaching, notably due to enhancing motivation achieved by employing approaches attracting student’s attention. Excelling aspects of audio visual presentation proved highly valuable particularly in classes with multi ethnic groups of students, as for example consistency between definitions and objects which were verbally and visually defined, eliminating possible misconceptions caused by mishearing or misinterpretation by the learner. Taking it all one step further as to how an educational system could be even more efficient, a new element would be needed revealing a credible judgment of learning scores and effectiveness of the learning process instantaneously as for example inner levels of activation and satisfaction. In fact, this could be made possible using existing technologies if subconscious neurophysiological responses of a learner could be ascertained and inferred to psycho-somatic conditions as they occur. A system including bio-sensing, data analysis and processing in real time able to provide quantified markers of psychosomatic states of a learner would help enormously in next generations of educational practice. Incorporating data of student engagement and active involvement could help to deduce the interest of a learner, which is known to improve sensitisation in implicit, incidental and also in classical learning. Experimental settings used in previous studies attempting to incorporate physiological responses and interpretations into responsive educational settings have faced major obstacles. Operational issues caused by the requirements of the devices used for the acquisition of physiological signals such as electrodes and movement restrictions have reduced the progress of such settings to laboratory environments. In such settings as described above, the effects of wiring harnesses and sensory components produced an additional psychological burden on the participants. Consequently, the need to approach the physiological data acquisition from a new angle with seamless and unnoticeable operation is apparent. The challenge to design, develop and validate a system that being minimally obstructive and literally unnoticed by the user would uncover combined subconscious expressions of a learner was the primary objective of this research. Physiological data of Heart Rate and Skin Trans-Conductance (Electro-dermal Response) elected as vitally important and highly appropriate to produce the input of data required to evaluate a behavioural concept model. The behavioural assessment model entailed vector classifiers producing directional interpretations of measurements. Directional information (Gradient response) has been derived by comparison of measurements to previously measured values in real time. Assessing the effectiveness and accuracy of the adopted model to deduce attention and engagement of a learner in real time formed the second major objective. For this purpose, a series of relevant experimental methodologies have been employed. Data produced using formal personality assessments have also been investigated in conjunction with those derived from physiological responses in order to identify personality related particularities. The final part of this work has been supplemented by propositions and suggestions with regards to various applications of the system in accomplishment of the initial aims

Exploring Preservice Teachers\u27 Affective Response to Disruptive Student Behavior in an Immersive Simulation Classroom

This mixed methods study investigated changes in preservice teachers\u27 affective response to disruptive student behavior within the TeachLivE, immersive simulation classroom. Preservice teachers completed two simulation teaching sessions, during which they were exposed to five different disruptive student behavior events in each. All teaching sessions were recorded and post-processed using iMotions Affectiva Affdex software to collect data on preservice teachers\u27 emotion expression and valence during their teaching experiences. At the end of each teaching session, participants completed a self-report survey on their level of stress. Simulated teaching sessions were followed-up with video stimulated recall sessions where participants reflected on their feelings during the simulation. The goal of this research was to examine changes in preservice teachers\u27 affective response to stress, with repeated exposure to disruptive student behavior, to determine if it had a desensitization effect, potentially increasing emotional regulation ability and decreasing negative emotional responses. Descriptive statistics were used to examine differences in emotional valence by disruptive student events and teaching sessions. Paired samples t-tests were conducted to examine if mean differences existed in self-reported stress within and between teaching sessions. Additional qualitative analysis of video stimulated recall sessions was conducted using thematic analysis. Analysis revealed minimal difference in preservice teachers\u27 positive or negative emotional valence in response to disruptive student behavior events within and between teaching sessions. There was a statistically significant change in self-reported stress from the first simulated teaching session to the second. Analysis of video stimulated recall reflections revealed themes of cognitive dissonance, behavior-induced stress, and difficulty with virtual behavior management

Modalities Matter

This thesis is a qualitative study that seeks to investigate pupils’ experiences, preferences, and attitudes when reading a fictional novel on different modalities. The motivation behind this study is a wish to spread the love of reading to a generation consisting of digital natives. We, therefore, wanted to explore how we could use fictional novels in the English classroom. A common thread that runs through our study is the focus on phenomenological immersion. We consider immersion to be the key element for a positive reading experience, and thus we have attempted to facilitate the reading experience for the pupils to enter this mindset. Our research was conducted in a 9th-grade classroom and lasted for four weeks. During these weeks, the pupils read the novel The Curious Incident of the Dog in the Night-Time by Mark Haddon (2003) by using the E-book, paper book, and audiobook. In order to answer our research question, we have collected data from observation, interviews, reflection logs, questionnaires, and our personal continuous reflection log. Our overall methodology was action research, which allowed us to implement changes and interventions we believed to improve the reading experience for the pupils. By following the development of each pupil’s reading experience, we have gotten an impression of how their mindsets change and are affected by the book being read, and the modality the book is read on. We have found that attitudes towards the book and comfortability with the modality are essential factors that affect the experience of reading a novel. Our study suggests that a reader may enter four different mindsets based on how they relate to these factors. If the reader is comfortable with the modality and has a positive attitude toward the book, the reader achieves immersion. If the reader is comfortable with the modality but has a negative attitude toward the book, the reader experiences boredom. If the reader is uncomfortable with the modality but has a positive attitude toward the book, the reader feels stressed. However, if the reader is uncomfortable with the modality and has a negative attitude toward the book, the reader gets disengaged. We have created a model called “The Mindset Map of Pupils’ Reading Experiences” to visualize this interplay of factors. Our intention for the model is in no way to label the pupils as one certain type of reader. On the contrary, we wish to visualize the dynamic process of reading, and how easily affected a reading experience can be

The Influence of Video Games on Adolescent Brain Activity

The current study examined electrical brain activation in adolescent participants playing three different video games. Forty-five school aged children (M=14.3 years, SD=1.5) were randomly assigned to play either a violent game, non-violent game, or a non-violent game specifically designed to train the brain. Electroencephalography (EEG) was recorded during video game play. Results revealed an asymmetric right hemisphere activation in the alpha band for participants in violent game group, while those in the non-violent groups exhibited left hemispheric activation. Greater right activation in emotion literature denotes signs of withdrawal or avoidance from undesired stimulus. Implications of this finding as well as other findings related to electrical brain activation during video game play is discussed further in the manuscript