Biosignal controlled recommendation in entertainment systems

With the explosive growth of the entertainment contents and the ubiquitous access of them via fixed or mobile computing devices, recommendation systems become essential tools to help the user to find the right entertainment at the right time and location. I envision that by integrating the bio signal input into the recommendation process, it will help the users not only to find interesting contents, but also to increase one’s comfort level by taking into account the biosginal feedback from the users. The goal of this project was to develop a biosignal controlled entertainment recommendation system that increases the user’s comfort level by reducing the level of stress. As the starting point, this project aims to contribute to the field of recommendation systems with two points. The first is the mechanism of embedding the biosignal non-intrusively into the recommendation process. The second is the strategy of the biosignal controlled recommendation to reduce stress. Heart rate controlled in-flight music recommendation is chosen as its application domain. The hypothesis of this application is that, the passenger's heart rate deviates from the normal due to unusual long haul flight cabin environment. By properly designing a music recommendation system to recommend heart rate controlled personalized music playlists to the passenger, the passengers' heart rate can be uplifted, down-lifted back to normal or kept within normal, thus their stress can be reduced. Four research questions have been formulated based on this hypothesis. After the literature study, the project went mainly through three phases: framework design, system implementation and user evaluation to answer these research questions. During the framework design phase, the heart rate was firstly modeled as the states of bradycardia, normal and tachycardia. The objective of the framework is that, if the user's heart rate is higher or lower than the normal heart rate, the system recommends a personalized music playlist accordingly to transfer the user’s heart rate back to normal, otherwise to keep it at normal. The adaptive framework integrates the concepts of context adaptive systems, user profiling, and the methods of using music to adjust the heart rate in a feedback control system. In the feedback loop, the playlists were composed using a Markov decision process. Yet, the framework allows the user to reject the recommendations and to manually select the favorite music items. During this process, the system logs the interactions between the user and the system for later learning the user’s latest music preferences. The designed framework was then implemented with platform independent software architecture. The architecture has five abstraction levels. The lowest resource level contains the music source, the heart rate sensors and the user profile information. The second layer is for resource management. In this layer are the manager components to manage the resources from the first layer and to modulate the access from upper layers to these resources. The third layer is the database, acting as a data repository. The fourth layer is for the adaptive control, which includes the user feedback log, the inference engine and the preference learning component. The top layer is the user interface. In this architecture, the layers and the components in the layers are loosely coupled, which ensures the flexibility. The implemented system was used in the user experiments to validate the hypothesis. The experiments simulated the long haul flights from Amsterdam to Shanghai with the same time schedule as the KLM flights. Twelve subjects were invited to participate in the experiments. Six were allocated to the controlled group and others were allocated to the treatment group. In addition to a normal entertainment system for the control group, the treatment group was also provided with the heart rate controlled music recommendation system. The experiments results validated the hypothesis and answered the research questions. The passenger's heart rate deviates from normal. In our user experiments, the passenger's heart rate was in the bradycardia state 24.6% of time and was in the tachycardia state 7.3% of time. The recommended uplifting music reduces the average bradycardia state duration from 14.78 seconds in the control group to 6.86 seconds in the treatment group. The recommended keeping music increases the average normal state duration from 24.66 seconds in the control group to 29.79 seconds in the treatment group. The recommended down-lifting music reduces the average tachycardia state duration from 13.89 seconds in the control group to 6.53 seconds in the treatment group. Compared to the control group, the stress of the treatment group has been reduced significantly

Psychological and psychophysiological effects of auditory and visual stimuli during various modes of exercise

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.This research programme had three principal objectives. First, to assess the stability of the exercise heart rate-music tempo preference relationship and its relevance to a range of psychological outcomes. Second, to explore the influence of two personal factors (motivational orientation and dominant attentional style) in a naturalistic exercise-to-music setting. Third, to examine means by which to enhance the exercise experience above and below the ventilatory threshold. In Study 1, a mixed-methods approach was employed to capture responses to differing music tempo conditions across a range of exercise intensities. Results in Study 1 did not support a cubic relationship (Karageorghis et al., 2011) but rather a quadratic one, and there was a weak association between the optimal choice of music tempo and positive psychological outcomes. Music conditions reduced the number of associative thoughts by ~10% across all exercise intensities. Study 2 employed questionnaires with a large sample of female participants (n = 417) attending exercise-to-music classes. Results indicate that motivational orientation and attentional style (Association vs. Dissociation) influence responses to an exercise-to-music class. Study 3 examined the effects of external stimuli (music and video) on psychological variables at moderate and high exercise intensities. Findings served to demonstrate that manipulations of attentional focus can have a salient influence on affect and enjoyment even during high-intensity exercise. The contributions of the research programme include providing empirical evidence that attention can be manipulated during high-intensity exercise using theoretically-guided music selections, and music in combination with video footage, which enhance the exercise experience. Further, the research programme advances understanding of how motivational orientation and attentional style influence responses to music during exercise

The psychophysiology primer : A guide to methods and a broad review with a focus on human-computer interaction

Publisher Copyright: © 2016 B. Cowley, M. Filetti, K. Lukander.Peer reviewe

Exploiting physiological changes during the flow experience for assessing virtual-reality game design.

Immersive experiences are considered the principal attraction of video games. Achieving a healthy balance between the game's demands and the user's skills is a particularly challenging goal. However, it is a coveted outcome, as it gives rise to the flow experience – a mental state of deep concentration and game engagement. When this balance fractures, the player may experience considerable disinclination to continue playing, which may be a product of anxiety or boredom. Thus, being able to predict manifestations of these psychological states in video game players is essential for understanding player motivation and designing better games. To this end, we build on earlier work to evaluate flow dynamics from a physiological perspective using a custom video game. Although advancements in this area are growing, there has been little consideration given to the interpersonal characteristics that may influence the expression of the flow experience. In this thesis, two angles are introduced that remain poorly understood. First, the investigation is contextualized in the virtual reality domain, a technology that putatively amplifies affective experiences, yet is still insufficiently addressed in the flow literature. Second, a novel analysis setup is proposed, whereby the recorded physiological responses and psychometric self-ratings are combined to assess the effectiveness of our game's design in a series of experiments. The analysis workflow employed heart rate and eye blink variability, and electroencephalography (EEG) as objective assessment measures of the game's impact, and self-reports as subjective assessment measures. These inputs were submitted to a clustering method, cross-referencing the membership of the observations with self-report ratings of the players they originated from. Next, this information was used to effectively inform specialized decoders of the flow state from the physiological responses. This approach successfully enabled classifiers to operate at high accuracy rates in all our studies. Furthermore, we addressed the compression of medium-resolution EEG sensors to a minimal set required to decode flow. Overall, our findings suggest that the approaches employed in this thesis have wide applicability and potential for improving game designing practices

Conception de système de traitement de données sur les émotions d’un être humain dans un environnement mobile et incertain

Une émotion humaine est considérée comme un état d’esprit d’un individu qui est complexe et intense, débutant de manière brutale et peut durer pendant une période relativement brève. Les émotions affectent généralement à la fois l’état physiologique et psychologique et peuvent aider à améliorer la santé humaine et l’efficacité au travail si elles sont positives, tandis que les émotions négatives peuvent causer des problèmes de santé et de comportements très graves. La détection et la surveillance des émotions sont primordiales dans de nombreux domaines tels que la conduite de véhicules, afin d’agir dans le temps opportun en cas de présence d’un état émotionnel négatif qui peut affecter dangereusement la vie du conducteur. Dans la science, on a défini plusieurs méthodes de détection d’émotions qui peuvent être classifiées en deux grandes catégories ; l’une utilise les signaux physiques humains tels que l’expression faciale, la parole, le geste, la posture, etc., qui ont l’avantage d’être facilement collectés et étudiés, mais qui souffrent d’une fiabilité modeste en raison de la possibilité de ne pas montrer les signaux physiques vrais pour cacher de véritables émotions. La deuxième catégorie utilise les signaux internes (les signaux physiologiques), qui comprennent l’électrocardiogramme (ECG), l’électroencéphalogramme (EEG), la température (T), l’électromyogramme (EMG), etc. qui sont plus fiable due à leur nature interne et non contrôler directement par l’être humain. Dans cette thèse, nous avons étudié le problème de la détection des émotions humaines chez un conducteur de véhicule en se basant sur le signal ECG. Pour cela, nous avons proposé trois contributions liées à la détection des émotions. La première est une approche d’optimisation des paramètres de classification des catégories des signaux ECG qui parmi elle une classe ECG anormale représentant un état émotionnel inhabituel. La deuxième contribution est une version améliorée de l’approche Random Forest pour la dé- tection de l’état du stress d’un conducteur. La troisième contribution est un système de détection d’émotions en suggérant une approche d’apprentissage profond ; il s’agit d’un nouveau réseau de ivneurones convolutif et d’augmentation de données qui considère la variabilité de la fréquence cardiaque (Heart Rate Variability -HRV-) comme critère essentiel de détection. Le système proposé a été bien développé et prouvé par une étude de validation et une comparaison avec les travaux de référence similaires proposés dans la littérature

Emotion and Stress Recognition Related Sensors and Machine Learning Technologies

This book includes impactful chapters which present scientific concepts, frameworks, architectures and ideas on sensing technologies and machine learning techniques. These are relevant in tackling the following challenges: (i) the field readiness and use of intrusive sensor systems and devices for capturing biosignals, including EEG sensor systems, ECG sensor systems and electrodermal activity sensor systems; (ii) the quality assessment and management of sensor data; (iii) data preprocessing, noise filtering and calibration concepts for biosignals; (iv) the field readiness and use of nonintrusive sensor technologies, including visual sensors, acoustic sensors, vibration sensors and piezoelectric sensors; (v) emotion recognition using mobile phones and smartwatches; (vi) body area sensor networks for emotion and stress studies; (vii) the use of experimental datasets in emotion recognition, including dataset generation principles and concepts, quality insurance and emotion elicitation material and concepts; (viii) machine learning techniques for robust emotion recognition, including graphical models, neural network methods, deep learning methods, statistical learning and multivariate empirical mode decomposition; (ix) subject-independent emotion and stress recognition concepts and systems, including facial expression-based systems, speech-based systems, EEG-based systems, ECG-based systems, electrodermal activity-based systems, multimodal recognition systems and sensor fusion concepts and (x) emotion and stress estimation and forecasting from a nonlinear dynamical system perspective