Affective automotive user interfaces

Technological progress in the fields of ubiquitous sensing and machine learning has been fueling the development of user-aware human-computer interaction in recent years. Especially natural user interfaces, like digital voice assistants, can benefit from understanding their users in order to provide a more naturalistic experience. Such systems can, for example, detect the emotional state of users and accordingly act in an empathic way. One major research field working on this topic is Affective Computing, where psycho-physiological measures, speech input, and facial expressions are used to sense human emotions. Affective data allows natural user interfaces to respond to emotions, providing promising perspectives not only for user experience design but also for safety aspects. In automotive environments, informed estimations of the driver’s state can potentially avoid dangerous errors and evoking positive emotions can improve the experience of driving. This dissertation explores Affective Automotive User Interfaces using two basic interaction paradigms: firstly, emotion regulation systems react to the current emotional state of the user based on live sensing data, allowing for quick interventions. Secondly, emotional interaction synthesizes experiences which resonate with the user on an emotional level. The constituted goals of these two interaction approaches are the promotion of safe behavior and an improvement of user experience. Promoting safe behavior through emotion regulation: Systems which detect and react to the driver’s state are expected to have great potential for improving road safety. This work presents a model and methods needed to investigate such systems and an exploration of several approaches to keep the driver in a safe state. The presented methods include techniques to induce emotions and to sample the emotional state of drivers. Three driving simulator studies investigate the impacts of emotionaware interventions in the form of implicit cues, visual mirroring and empathic speech synthesis. We envision emotion-awareness as a safety feature which can detect if a driver is unfit or in need of support, based on the propagation of robust emotion detection technology. Improving user experience with emotional interaction: Emotional perception is an essential part of user experience. This thesis entails methods to build emotional experiences derived from a variety of lab and simulator studies, expert feedback, car-storming sessions and design thinking workshops. Systems capable of adapting to the user’s preferences and traits in order to create an emotionally satisfactory user experience do not require the input of emotion detection. They rather create value through general knowledge about the user by adapting the output they generate. During this research, cultural and generational influences became evident, which have to be considered when implementing affective automotive user interfaces in future cars. We argue that the future of user-aware interaction lies in adapting not only to the driver’s preferences and settings but also to their current state. This paves the way for the regulation of safe behavior, especially in safety-critical environments like cars, and an improvement of the driving experience.Aktuelle Fortschritte in den Bereichen des Machine Learning und Ubiquitous Computing ermöglichen es heute adaptive Mensch-Maschine-Schnittstellen zu realisieren. Vor allem natürliche Interaktion, wie wir sie von Sprachassistenten kennen, profitiert von einem verbesserten Verständnis des Nutzerverhaltens. Zum Beispiel kann ein Assistent mit Informationen über den emotionalen Zustand des Nutzers natürlicher interagieren, vielleicht sogar Empathie zeigen. Affective Computing ist das damit verbundene Forschungsfeld, das sich damit beschäftigt menschliche Emotionen durch Beobachtung von physiologischen Daten, Sprache und Mimik zu erkennen. Dabei ermöglicht Emotionserkennung natürliche Interaktion auf Basis des Fahrer/innenzustands, was nicht nur vielversprechend in Bezug auf die Gestaltung des Nutzerelebnisses klingt, sondern auch Anwendungen im Bereich der Verkehrssicherheit hat. Ein Einsatz im Fahrkontext könnte so vermeidbare Unfälle verringern und gleichzeitig Fahrer durch emotionale Interaktion begeistern. Diese Dissertation beleuchtet Affective Automotive User Interfaces – zu Deutsch in etwa Emotionsadaptive Benutzerschnittstellen im Fahrzeug – auf Basis zweier inhaltlicher Säulen: erstens benutzen wir Ansätze zur Emotionsregulierung, um im Falle gefährlicher Fahrerzustände einzugreifen. Zweitens erzeugen wir emotional aufgeladene Interaktionen, um das Nutzererlebnis zu verbessern. Erhöhte Sicherheit durch Emotionsregulierung: Emotionsadaptiven Systemen wird ein großes Potenzial zur Verbesserung der Verkehrssicherheit zugeschrieben. Wir stellen ein Modell und Methoden vor, die zur Untersuchung solcher Systeme benötigt werden und erforschen Ansätze, die dazu dienen Fahrer in einer Gefühlslage zu halten, die sicheres Handeln erlaubt. Die vorgestellten Methoden beinhalten Ansätze zur Emotionsinduktion und -erkennung, sowie drei Fahrsimulatorstudien zur Beeinflussung von Fahrern durch indirekte Reize, Spiegeln von Emotionen und empathischer Sprachinteraktion. Emotionsadaptive Sicherheitssysteme können in Zukunft beeinträchtigten Fahrern Unterstützung leisten und so den Verkehr sicherer machen, vorausgesetzt die technischen Grundlagen der Emotionserkennung gewinnen an Reife. Verbesserung des Nutzererlebnisses durch emotionale Interaktion: Emotionen tragen einen großen Teil zum Nutzerlebnis bei, darum ist es nur sinnvoll den zweiten Fokuspunkt dieser Arbeit auf systeminitiierte emotionale Interaktion zu legen.Wir stellen die Ergebnisse nutzerzentrierter Ideenfindung und mehrer Evaluationsstudien der resultierenden Systeme vor. Um sich den Vorlieben und Eigenschaften von Nutzern anzupassen wird nicht zwingend Emotionserkennung benötigt. Der Mehrwert solcher Systeme besteht vielmehr darin, auf Basis verfügbarer Verhaltensdaten ein emotional anspruchsvolles Erlebnis zu ermöglichen. In unserer Arbeit stoßen wir außerdem auf kulturelle und demografische Einflüsse, die es bei der Gestaltung von emotionsadaptiven Nutzerschnittstellen zu beachten gibt. Wir sehen die Zukunft nutzeradaptiver Interaktion im Fahrzeug nicht in einer rein verhaltensbasierten Anpassung, sondern erwarten ebenso emotionsbezogene Innovationen. Dadurch können zukünftige Systeme sicherheitsrelevantes Verhalten regulieren und gleichzeitig das Fortbestehen der Freude am Fahren ermöglichen

Affective automotive user interfaces

Technological progress in the fields of ubiquitous sensing and machine learning has been fueling the development of user-aware human-computer interaction in recent years. Especially natural user interfaces, like digital voice assistants, can benefit from understanding their users in order to provide a more naturalistic experience. Such systems can, for example, detect the emotional state of users and accordingly act in an empathic way. One major research field working on this topic is Affective Computing, where psycho-physiological measures, speech input, and facial expressions are used to sense human emotions. Affective data allows natural user interfaces to respond to emotions, providing promising perspectives not only for user experience design but also for safety aspects. In automotive environments, informed estimations of the driver’s state can potentially avoid dangerous errors and evoking positive emotions can improve the experience of driving. This dissertation explores Affective Automotive User Interfaces using two basic interaction paradigms: firstly, emotion regulation systems react to the current emotional state of the user based on live sensing data, allowing for quick interventions. Secondly, emotional interaction synthesizes experiences which resonate with the user on an emotional level. The constituted goals of these two interaction approaches are the promotion of safe behavior and an improvement of user experience. Promoting safe behavior through emotion regulation: Systems which detect and react to the driver’s state are expected to have great potential for improving road safety. This work presents a model and methods needed to investigate such systems and an exploration of several approaches to keep the driver in a safe state. The presented methods include techniques to induce emotions and to sample the emotional state of drivers. Three driving simulator studies investigate the impacts of emotionaware interventions in the form of implicit cues, visual mirroring and empathic speech synthesis. We envision emotion-awareness as a safety feature which can detect if a driver is unfit or in need of support, based on the propagation of robust emotion detection technology. Improving user experience with emotional interaction: Emotional perception is an essential part of user experience. This thesis entails methods to build emotional experiences derived from a variety of lab and simulator studies, expert feedback, car-storming sessions and design thinking workshops. Systems capable of adapting to the user’s preferences and traits in order to create an emotionally satisfactory user experience do not require the input of emotion detection. They rather create value through general knowledge about the user by adapting the output they generate. During this research, cultural and generational influences became evident, which have to be considered when implementing affective automotive user interfaces in future cars. We argue that the future of user-aware interaction lies in adapting not only to the driver’s preferences and settings but also to their current state. This paves the way for the regulation of safe behavior, especially in safety-critical environments like cars, and an improvement of the driving experience.Aktuelle Fortschritte in den Bereichen des Machine Learning und Ubiquitous Computing ermöglichen es heute adaptive Mensch-Maschine-Schnittstellen zu realisieren. Vor allem natürliche Interaktion, wie wir sie von Sprachassistenten kennen, profitiert von einem verbesserten Verständnis des Nutzerverhaltens. Zum Beispiel kann ein Assistent mit Informationen über den emotionalen Zustand des Nutzers natürlicher interagieren, vielleicht sogar Empathie zeigen. Affective Computing ist das damit verbundene Forschungsfeld, das sich damit beschäftigt menschliche Emotionen durch Beobachtung von physiologischen Daten, Sprache und Mimik zu erkennen. Dabei ermöglicht Emotionserkennung natürliche Interaktion auf Basis des Fahrer/innenzustands, was nicht nur vielversprechend in Bezug auf die Gestaltung des Nutzerelebnisses klingt, sondern auch Anwendungen im Bereich der Verkehrssicherheit hat. Ein Einsatz im Fahrkontext könnte so vermeidbare Unfälle verringern und gleichzeitig Fahrer durch emotionale Interaktion begeistern. Diese Dissertation beleuchtet Affective Automotive User Interfaces – zu Deutsch in etwa Emotionsadaptive Benutzerschnittstellen im Fahrzeug – auf Basis zweier inhaltlicher Säulen: erstens benutzen wir Ansätze zur Emotionsregulierung, um im Falle gefährlicher Fahrerzustände einzugreifen. Zweitens erzeugen wir emotional aufgeladene Interaktionen, um das Nutzererlebnis zu verbessern. Erhöhte Sicherheit durch Emotionsregulierung: Emotionsadaptiven Systemen wird ein großes Potenzial zur Verbesserung der Verkehrssicherheit zugeschrieben. Wir stellen ein Modell und Methoden vor, die zur Untersuchung solcher Systeme benötigt werden und erforschen Ansätze, die dazu dienen Fahrer in einer Gefühlslage zu halten, die sicheres Handeln erlaubt. Die vorgestellten Methoden beinhalten Ansätze zur Emotionsinduktion und -erkennung, sowie drei Fahrsimulatorstudien zur Beeinflussung von Fahrern durch indirekte Reize, Spiegeln von Emotionen und empathischer Sprachinteraktion. Emotionsadaptive Sicherheitssysteme können in Zukunft beeinträchtigten Fahrern Unterstützung leisten und so den Verkehr sicherer machen, vorausgesetzt die technischen Grundlagen der Emotionserkennung gewinnen an Reife. Verbesserung des Nutzererlebnisses durch emotionale Interaktion: Emotionen tragen einen großen Teil zum Nutzerlebnis bei, darum ist es nur sinnvoll den zweiten Fokuspunkt dieser Arbeit auf systeminitiierte emotionale Interaktion zu legen.Wir stellen die Ergebnisse nutzerzentrierter Ideenfindung und mehrer Evaluationsstudien der resultierenden Systeme vor. Um sich den Vorlieben und Eigenschaften von Nutzern anzupassen wird nicht zwingend Emotionserkennung benötigt. Der Mehrwert solcher Systeme besteht vielmehr darin, auf Basis verfügbarer Verhaltensdaten ein emotional anspruchsvolles Erlebnis zu ermöglichen. In unserer Arbeit stoßen wir außerdem auf kulturelle und demografische Einflüsse, die es bei der Gestaltung von emotionsadaptiven Nutzerschnittstellen zu beachten gibt. Wir sehen die Zukunft nutzeradaptiver Interaktion im Fahrzeug nicht in einer rein verhaltensbasierten Anpassung, sondern erwarten ebenso emotionsbezogene Innovationen. Dadurch können zukünftige Systeme sicherheitsrelevantes Verhalten regulieren und gleichzeitig das Fortbestehen der Freude am Fahren ermöglichen

On driver behavior recognition for increased safety:A roadmap

Advanced Driver-Assistance Systems (ADASs) are used for increasing safety in the automotive domain, yet current ADASs notably operate without taking into account drivers’ states, e.g., whether she/he is emotionally apt to drive. In this paper, we first review the state-of-the-art of emotional and cognitive analysis for ADAS: we consider psychological models, the sensors needed for capturing physiological signals, and the typical algorithms used for human emotion classification. Our investigation highlights a lack of advanced Driver Monitoring Systems (DMSs) for ADASs, which could increase driving quality and security for both drivers and passengers. We then provide our view on a novel perception architecture for driver monitoring, built around the concept of Driver Complex State (DCS). DCS relies on multiple non-obtrusive sensors and Artificial Intelligence (AI) for uncovering the driver state and uses it to implement innovative Human–Machine Interface (HMI) functionalities. This concept will be implemented and validated in the recently EU-funded NextPerception project, which is briefly introduced

Affective responses to chromatic ambient light in a vehicle

This study investigates the emotional responses to the color of vehicle interior lighting using self-assessment and electroencephalography (EEG). The study was divided into two sessions: the first session investigated the potential of ambient lighting colors, and the second session was used to develop in-vehicle lighting color guidelines. Every session included thirty subjects. In the first session, four lighting colors were assessed using seventeen adjectives. As a result, 'Preference, Softness, Brightness, and Uniqueness were found to be the four factors that best characterize the atmospheric properties of interior lighting in vehicles. Ambient illumination, according to EEG data, increased people's arousal and lowered their alpha waves. The following session investigated a wider spectrum of colors using four factors extracted from the previous session. As a result, bluish and purplish lighting colors had the highest preference and uniqueness among ten lighting colors. Green received an intermediate preference and a high uniqueness score. With its great brightness and softness, Neutral White also achieved an intermediate preference rating. Despite receiving a low preference rating, warm colors were considered to be soft. Red was the least preferred color, but its uniqueness and roughness were highly rated. This study is expected to provide a basic theory on emotional lighting guidelines in the vehicle context, providing manufacturers with objective rationale

“Play Your Anger”: A report on the empathic in-vehicle interface workshop

Empathic in-vehicle interfaces are critical in improving user safety and experiences. There has been much research on how to estimate drivers’ affective states, whereas little research has investigated intervention methods that mitigate potential impacts from the driver’s affective states on their driving performance and user experiences. To enhance the development of in-vehicle interfaces considering emotional aspects, we have organized a workshop series to gather automotive user interface experts to discuss this topic at the International ACM Conference on Automotive User Interfaces and Interactive Vehicular Applications (AutoUI). The present paper focuses particularly on the intervention methods created by the experts and proposes design recommendations for future empathic in-vehicle interfaces. We hope this work can spark lively discussions on the importance of drivers’ affective states in their user experience of automated vehicles and pose the right direction

What If Your Car Would Care? Exploring Use Cases For Affective Automotive User Interfaces

In this paper we present use cases for affective user interfaces (UIs) in cars and how they are perceived by potential users in China and Germany. Emotion-aware interaction is enabled by the improvement of ubiquitous sensing methods and provides potential benefits for both traffic safety and personal well-being. To promote the adoption of affective interaction at an international scale, we developed 20 mobile in-car use cases through an inter-cultural design approach and evaluated them with 65 drivers in Germany and China. Our data shows perceived benefits in specific areas of pragmatic quality as well as cultural differences, especially for socially interactive use cases. We also discuss general implications for future affective automotive UI. Our results provide a perspective on cultural peculiarities and a concrete starting point for practitioners and researchers working on emotion-aware interfaces

Personal informatics and negative emotions during commuter driving:Effects of data visualization on cardiovascular reactivity & mood

Mobile technology and wearable sensors can provide objective measures of psychological stress in everyday life. Data from sensors can be visualized and viewed by the user to increase self-awareness and promote adaptive coping strategies. A capacity to effectively self-regulate negative emotion can mitigate the biological process of inflammation, which has implications for long-term health. Two studies were undertaken utilizing a mobile lifelogging platform to collect cardiovascular data over a week of real-life commuter driving. The first was designed to establish a link between cardiovascular markers of inflammation and the experience of anger during commuter driving in the real world. Results indicated that an ensemble classification model provided an accuracy rate of 73.12% for the binary classification of episodes of high vs. low anger based upon a combination of features derived from driving (e.g. vehicle speed) and cardiovascular psychophysiology (heart rate, heart rate variability, pulse transit time). During the second study, participants interacted with an interactive, geolocated visualisation of vehicle parameters, photographs and cardiovascular psychophysiology collected over two days of commuter driving (pre-test). Data were subsequently collected over two days of driving following their interaction with the dynamic, data visualization (post-test). A comparison of pre- and post-test data revealed that heart rate significantly reduced during episodes of journey impedance after interaction with the data visualization. There was also evidence that heart rate variability increased during the post-test phase, suggesting greater vagal activation and adaptive coping. Subjective mood data were collected before and after each journey, but no statistically significant differences were observed between pre- and post-test periods. The implications of both studies for ambulatory monitoring, user interaction and the capacity of personal informatics to enhance long-term health are discussed

Design and Development of a Real-Time Bio-Sensing System Assessing Student Mental Workload and Engagement

Ο εντοπισμός του επακριβούς επιπέδου προσήλωσης και εμπλοκής των μαθητών με το περιεχόμενο διδασκαλίας στην τάξη είναι ένας από τους πιο μεγαλεπήβολους στόχους των ερευνητών της εκπαιδευτικής και επιστημονικής κοινότητας. (Lang, 1995, Grossberg, 1987). Σχετικές διεπιστημονικές ερευνητικές προσπάθειες προσαύξησης ενδιαφέροντος και εντοπισμού της αποτελεσματικότητας των διδακτικών πρακτικών βασίζονται σε τυπικές μελέτες από τον χώρο της ψυχολογίας, της παιδαγωγικής, της παιδοψυχολογίας και της ψυχοφυσιολογίας. Νέες τεχνολογίες έχουν εισάγει διαγνωστικές συσκευές δανεισμένες από τον χώρο της ιατρικής με σκοπό να εκμεταλλευτούν τις δυνατότητες μετρήσεων βιολογικών σημάτων τα οποία αποτελούν επιβεβαιωμένες εκφράσεις ψυχοφυσιολογικών καταστάσεων οι οποίες μπορούν να μεταφραστούν σε εκδηλώσεις διέγερσης και διάθεσης. Οι ιατρικές συσκευές απαιτούν εργαστηριακό περιβάλλον λόγω των αναγκών χρήσης ηλεκτροδίων, κινητικών περιορισμών, συγχρονισμού και ομοιομορφίας των στοιχείων που προκύπτουν και γι’ αυτό τον λόγο δεν μπόρεσαν ποτέ να αποδόσουν μια προσιτή λύση εφαρμόσιμη ευρύτερα σε εκπαιδευτικό περιβάλλον. Στην παρούσα μελέτη, αναλύονται οι επιδόσεις μιας ειδικά κατασκευασμένης ηλεκτρονικής συσκευής, σχεδιασμένης ώστε να εξεταστούν οι δυνατότητες να εξαχθούν δείκτες ψυχοσωματικών εκφράσεων του χρήστη, με την δυνατότητα να χρησιμοποιείται εύχρηστα στην τάξη χωρίς ηλεκτρόδια και επηρεασμούς από προσαρτήσεις. Το ολοκληρωμένο σύστημα μέτρησης και αποτύπωσης συμπερασμάτων είναι βασισμένο σε μοντελοποίηση συμπεριφορών αλλαγής του καρδιακού παλμού και της ειδικής διηλεκτρικής αγωγιμότητας του δέρματος σε πραγματικό χρόνο. Η συσκευή χρησιμοποιεί οπτικούς και διηλεκτρικούς αισθητήρες επαφής και έχει μελετηθεί σε αντιπαραβολή με διαβαθμισμένα περιβάλλοντα προκλητών καταστάσεων νοητικής φόρτισης. Σειρές πειραματικών διαδικασιών εφαρμοσμένες σε διαβαθμισμένα σενάρια πρόκλησης ψυχοσωματικών διεγέρσεων έχουν ολοκληρωθεί για επικύρωση, μελέτη επιδόσεων και λειτουργία του συστήματος ακόμη και σε σύγκριση με εμπορικό προϊόν. Πειραματικά αποτελέσματα δείχνουν αξιόλογους συσχετισμούς του μοντέλου και των επιδόσεων του συστήματος με τις αναμενόμενες αποκρίσεις με ενθαρρυντικά ποσοστά ακρίβειας.Facing the challenge of improving adaptive interaction in educational technologies scientists and educators have turned their focal point in diverse areas ranging from educational, teaching and behavioural psychology to cognitive, affective and perceptual neuroscience. The introduction of digital technologies and interactive media tools in education has shown improved learning efficiency, much higher memory activation and assimilation than verbal teaching, notably due to enhancing motivation achieved by employing approaches attracting student’s attention. Excelling aspects of audio visual presentation proved highly valuable particularly in classes with multi ethnic groups of students, as for example consistency between definitions and objects which were verbally and visually defined, eliminating possible misconceptions caused by mishearing or misinterpretation by the learner. Taking it all one step further as to how an educational system could be even more efficient, a new element would be needed revealing a credible judgment of learning scores and effectiveness of the learning process instantaneously as for example inner levels of activation and satisfaction. In fact, this could be made possible using existing technologies if subconscious neurophysiological responses of a learner could be ascertained and inferred to psycho-somatic conditions as they occur. A system including bio-sensing, data analysis and processing in real time able to provide quantified markers of psychosomatic states of a learner would help enormously in next generations of educational practice. Incorporating data of student engagement and active involvement could help to deduce the interest of a learner, which is known to improve sensitisation in implicit, incidental and also in classical learning. Experimental settings used in previous studies attempting to incorporate physiological responses and interpretations into responsive educational settings have faced major obstacles. Operational issues caused by the requirements of the devices used for the acquisition of physiological signals such as electrodes and movement restrictions have reduced the progress of such settings to laboratory environments. In such settings as described above, the effects of wiring harnesses and sensory components produced an additional psychological burden on the participants. Consequently, the need to approach the physiological data acquisition from a new angle with seamless and unnoticeable operation is apparent. The challenge to design, develop and validate a system that being minimally obstructive and literally unnoticed by the user would uncover combined subconscious expressions of a learner was the primary objective of this research. Physiological data of Heart Rate and Skin Trans-Conductance (Electro-dermal Response) elected as vitally important and highly appropriate to produce the input of data required to evaluate a behavioural concept model. The behavioural assessment model entailed vector classifiers producing directional interpretations of measurements. Directional information (Gradient response) has been derived by comparison of measurements to previously measured values in real time. Assessing the effectiveness and accuracy of the adopted model to deduce attention and engagement of a learner in real time formed the second major objective. For this purpose, a series of relevant experimental methodologies have been employed. Data produced using formal personality assessments have also been investigated in conjunction with those derived from physiological responses in order to identify personality related particularities. The final part of this work has been supplemented by propositions and suggestions with regards to various applications of the system in accomplishment of the initial aims

Visual Anxiolytics: developing theory and design guidelines for abstract affective visualizations aimed at alleviating episodes of anxiety

Visual Anxiolytics is a novel term proposed to describe affective visualizations of which affective quality is predetermined and designed to alleviate anxiety and anxious pathology. This thesis presents ground theory and visual guidelines to inform the design of screen-based interfaces to give users aspects of a restorative and anxiolytic environment at a time when attention restoration is least likely and anxiety highly probable; during sedentary screen-time. Visual Anxiolytics are introduced as an affective layer of the interface capable of communicating affect through aesthetic, abstract, ambient emotion visualizations existing in the periphery of the screen and users’ vision. Their theory is brought into the field of Visual Communication Design from a number of disciplines; primarily Affective Computing, Human-Computer Interaction, Psychology, and Neuroscience. Visual Anxiolytics attempt to alleviate anxiety through restoration of attentional cognitive resources by rendering the digital environment restorative and by elicitation of positive emotions through affect communication. Design guidelines analyse and describe properties of anxiolytic affective visual attributes color, shape, motion, and visual depth, as well as compositional characteristics of Visual Anxiolytics. Potential implications for future research in emotion visualization and affect communication are discussed