Assessing the Effectiveness of Automated Emotion Recognition in Adults and Children for Clinical Investigation

Recent success stories in automated object or face recognition, partly fuelled by deep learning artificial neural network (ANN) architectures, has led to the advancement of biometric research platforms and, to some extent, the resurrection of Artificial Intelligence (AI). In line with this general trend, inter-disciplinary approaches have taken place to automate the recognition of emotions in adults or children for the benefit of various applications such as identification of children emotions prior to a clinical investigation. Within this context, it turns out that automating emotion recognition is far from being straight forward with several challenges arising for both science(e.g., methodology underpinned by psychology) and technology (e.g., iMotions biometric research platform). In this paper, we present a methodology, experiment and interesting findings, which raise the following research questions for the recognition of emotions and attention in humans: a) adequacy of well-established techniques such as the International Affective Picture System (IAPS), b) adequacy of state-of-the-art biometric research platforms, c) the extent to which emotional responses may be different among children or adults. Our findings and first attempts to answer some of these research questions, are all based on a mixed sample of adults and children, who took part in the experiment resulting into a statistical analysis of numerous variables. These are related with, both automatically and interactively, captured responses of participants to a sample of IAPS pictures

Brain Music : Sistema generativo para la creación de música simbólica a partir de respuestas neuronales afectivas

gráficas, tablasEsta tesis de maestría presenta una metodología de aprendizaje profundo multimodal innovadora que fusiona un modelo de clasificación de emociones con un generador musical, con el propósito de crear música a partir de señales de electroencefalografía, profundizando así en la interconexión entre emociones y música. Los resultados alcanzan tres objetivos específicos: Primero, ya que el rendimiento de los sistemas interfaz cerebro-computadora varía considerablemente entre diferentes sujetos, se introduce un enfoque basado en la transferencia de conocimiento entre sujetos para mejorar el rendimiento de individuos con dificultades en sistemas de interfaz cerebro-computadora basados en el paradigma de imaginación motora. Este enfoque combina datos de EEG etiquetados con datos estructurados, como cuestionarios psicológicos, mediante un método de "Kernel Matching CKA". Utilizamos una red neuronal profunda (Deep&Wide) para la clasificación de la imaginación motora. Los resultados destacan su potencial para mejorar las habilidades motoras en interfaces cerebro-computadora. Segundo, proponemos una técnica innovadora llamada "Labeled Correlation Alignment"(LCA) para sonificar respuestas neurales a estímulos representados en datos no estructurados, como música afectiva. Esto genera características musicales basadas en la actividad cerebral inducida por las emociones. LCA aborda la variabilidad entre sujetos y dentro de sujetos mediante el análisis de correlación, lo que permite la creación de envolventes acústicos y la distinción entre diferente información sonora. Esto convierte a LCA en una herramienta prometedora para interpretar la actividad neuronal y su reacción a estímulos auditivos. Finalmente, en otro capítulo, desarrollamos una metodología de aprendizaje profundo de extremo a extremo para generar contenido musical MIDI (datos simbólicos) a partir de señales de actividad cerebral inducidas por música con etiquetas afectivas. Esta metodología abarca el preprocesamiento de datos, el entrenamiento de modelos de extracción de características y un proceso de emparejamiento de características mediante Deep Centered Kernel Alignment, lo que permite la generación de música a partir de señales EEG. En conjunto, estos logros representan avances significativos en la comprensión de la relación entre emociones y música, así como en la aplicación de la inteligencia artificial en la generación musical a partir de señales cerebrales. Ofrecen nuevas perspectivas y herramientas para la creación musical y la investigación en neurociencia emocional. Para llevar a cabo nuestros experimentos, utilizamos bases de datos públicas como GigaScience, Affective Music Listening y Deap Dataset (Texto tomado de la fuente)This master’s thesis presents an innovative multimodal deep learning methodology that combines an emotion classification model with a music generator, aimed at creating music from electroencephalography (EEG) signals, thus delving into the interplay between emotions and music. The results achieve three specific objectives: First, since the performance of brain-computer interface systems varies significantly among different subjects, an approach based on knowledge transfer among subjects is introduced to enhance the performance of individuals facing challenges in motor imagery-based brain-computer interface systems. This approach combines labeled EEG data with structured information, such as psychological questionnaires, through a "Kernel Matching CKA"method. We employ a deep neural network (Deep&Wide) for motor imagery classification. The results underscore its potential to enhance motor skills in brain-computer interfaces. Second, we propose an innovative technique called "Labeled Correlation Alignment"(LCA) to sonify neural responses to stimuli represented in unstructured data, such as affective music. This generates musical features based on emotion-induced brain activity. LCA addresses variability among subjects and within subjects through correlation analysis, enabling the creation of acoustic envelopes and the distinction of different sound information. This makes LCA a promising tool for interpreting neural activity and its response to auditory stimuli. Finally, in another chapter, we develop an end-to-end deep learning methodology for generating MIDI music content (symbolic data) from EEG signals induced by affectively labeled music. This methodology encompasses data preprocessing, feature extraction model training, and a feature matching process using Deep Centered Kernel Alignment, enabling music generation from EEG signals. Together, these achievements represent significant advances in understanding the relationship between emotions and music, as well as in the application of artificial intelligence in musical generation from brain signals. They offer new perspectives and tools for musical creation and research in emotional neuroscience. To conduct our experiments, we utilized public databases such as GigaScience, Affective Music Listening and Deap DatasetMaestríaMagíster en Ingeniería - Automatización IndustrialInvestigación en Aprendizaje Profundo y señales BiológicasEléctrica, Electrónica, Automatización Y Telecomunicaciones.Sede Manizale

Calming effect of Clinically Designed Improvisatory Music for patients admitted to the epilepsy monitoring unit during the COVID-19 pandemic: a pilot study

BackgroundEpilepsy monitoring requires simulating seizure-inducing conditions which frequently causes discomfort to epilepsy monitoring unit (EMU) patients. COVID-19 hospital restrictions added another layer of stress during hospital admissions. The purpose of this pilot study was to provide evidence that live virtual Clinically Designed Improvisatory Music (CDIM) brings relief to EMU patients for their psychological distress.MethodsFive persons with epilepsy (PWEs) in the EMU during the COVID-19 lockdown participated in the study (average age ± SD = 30.2 ± 6 years). Continuous electroencephalogram (EEG) and electrocardiogram (EKG) were obtained before, during, and after live virtual CDIM. CDIM consisted of 40 minutes of calming music played by a certified clinical music practitioner (CMP) on viola. Post-intervention surveys assessed patients’ emotional state on a 1–10 Likert scale. Alpha/beta power spectral density ratio was calculated for each subject across the brain and was evaluated using one-way repeated analysis of variance, comparing 20 minutes before, during, and 20 minutes after CDIM. Post-hoc analysis was performed using paired t-test at the whole brain level and regions with peak changes.ResultsPatients reported enhanced emotional state (9 ± 1.26), decrease in tension (9.6 ± 0.49), decreased restlessness (8.6 ± 0.80), increased pleasure (9.2 ± 0.98), and likelihood to recommend (10 ± 0) on a 10-point Likert scale. Based on one-way repeated analysis of variance, alpha/beta ratio increased at whole-brain analysis (F3,12 = 5.01, P = 0.018) with a peak in midline (F3,12 = 6.63, P = 0.0068 for Cz) and anterior medial frontal region (F3,12 = 6.45, P = 0.0076 for Fz) during CDIM and showed a trend to remain increased post-intervention.ConclusionIn this pilot study, we found positive effects of CDIM as reported by patients, and an increased alpha/beta ratio with meaningful electroencephalographic correlates due to the calming effects in response to CDIM. Our study provides proof of concept that live virtual CDIM offered demonstrable comfort with biologic correlations for patients admitted in the EMU during the COVID-19 pandemic

Music, emotion and brain activity

A música e a emoção são conceitos complexos e as suas relações apresentam diferentes expressões no funcionamento cerebral que podem ser registadas através da realização do eletroencefalograma,permitindo o seu estudo e avaliação. Com o objetivo de demonstrar que a música e a emoção estão interrelacionadas e que influenciam a atividade cerebral, foi realizada a recolha de 10 artigos com datas de publicaçãocompreendidas entre 2015 e 2021 através da pesquisa científica em plataformas reconhecidas. Foram retiradas as principais informações de cada um dos estudos e relacionadas entre si. Verificou-se que para além de áreas e localizações em comum aquando do processamento de emoções induzidas pela música no cérebro, foram detetadas semelhanças nas características eletroencefalográficas registadas nos diferentes artigos analisados, mas também disparidades, especialmente quando consideradas influências indissociáveis como o gosto e a familiaridade.Abstract: Music and emotion are complex concepts and their relationships present different expressions in brain function that can be registered through the performance of the electroencephalogram, allowing its study and evaluation. In order to demonstrate that music and emotion are interrelated and that they influence brain activity, the compilation of 10 articles with publication dates between 2015 and 2021 was carried out through scientific research on well-known platforms. The main information from each of the studies has been collected and related to each other. It was noticed that besides areas and locations in common when processing emotions induced by music in the brain, similarities were detected in the electroencephalographi characteristics recorded in the different articles analysed, but also disparities, especially when considering inseparable influences such as taste and familiarity.info:eu-repo/semantics/publishedVersio

Modelling human emotions using immersive virtual reality, physiological signals and behavioural responses

Tesis por compendio[ES] El uso de la realidad virtual (RV) se ha incrementado notablemente en la comunidad científica para la investigación del comportamiento humano. En particular, la RV inmersiva ha crecido debido a la democratización de las gafas de realidad virtual o head mounted displays (HMD), que ofrecen un alto rendimiento con una inversión económica. Uno de los campos que ha emergido con fuerza en la última década es el Affective Computing, que combina psicofisiología, informática, ingeniería biomédica e inteligencia artificial, desarrollando sistemas que puedan reconocer emociones automáticamente. Su progreso es especialmente importante en el campo de la investigación del comportamiento humano, debido al papel fundamental que las emociones juegan en muchos procesos psicológicos como la percepción, la toma de decisiones, la creatividad, la memoria y la interacción social. Muchos estudios se han centrado en intentar obtener una metodología fiable para evocar y automáticamente identificar estados emocionales, usando medidas fisiológicas objetivas y métodos de aprendizaje automático. Sin embargo, la mayoría de los estudios previos utilizan imágenes, audios o vídeos para generar los estados emocionales y, hasta donde llega nuestro conocimiento, ninguno de ellos ha desarrollado un sistema de reconocimiento emocional usando RV inmersiva. Aunque algunos trabajos anteriores sí analizan las respuestas fisiológicas en RV inmersivas, estos no presentan modelos de aprendizaje automático para procesamiento y clasificación automática de bioseñales. Además, un concepto crucial cuando se usa la RV en investigación del comportamiento humano es la validez: la capacidad de evocar respuestas similares en un entorno virtual a las evocadas por el espacio físico. Aunque algunos estudios previos han usado dimensiones psicológicas y cognitivas para comparar respuestas entre entornos reales y virtuales, las investigaciones que analizan respuestas fisiológicas o comportamentales están mucho menos extendidas. Según nuestros conocimientos, este es el primer trabajo que compara entornos físicos con su réplica en RV, empleando respuestas fisiológicas y algoritmos de aprendizaje automático y analizando la capacidad de la RV de transferir y extrapolar las conclusiones obtenidas al entorno real que se está simulando. El objetivo principal de la tesis es validar el uso de la RV inmersiva como una herramienta de estimulación emocional usando respuestas psicofisiológicas y comportamentales en combinación con algoritmos de aprendizaje automático, así como realizar una comparación directa entre un entorno real y virtual. Para ello, se ha desarrollado un protocolo experimental que incluye entornos emocionales 360º, un museo real y una virtualización 3D altamente realista del mismo museo. La tesis presenta novedosas contribuciones del uso de la RV inmersiva en la investigación del comportamiento humano, en particular en lo relativo al estudio de las emociones. Esta ayudará a aplicar metodologías a estímulos más realistas para evaluar entornos y situaciones de la vida diaria, superando las actuales limitaciones de la estimulación emocional que clásicamente ha incluido imágenes, audios o vídeos. Además, en ella se analiza la validez de la RV realizando una comparación directa usando una simulación altamente realista. Creemos que la RV inmersiva va a revolucionar los métodos de estimulación emocional en entornos de laboratorio. Además, su sinergia junto a las medidas fisiológicas y las técnicas de aprendizaje automático, impactarán transversalmente en muchas áreas de investigación como la arquitectura, la salud, la evaluación psicológica, el entrenamiento, la educación, la conducción o el marketing, abriendo un nuevo horizonte de oportunidades para la comunidad científica. La presente tesis espera contribuir a caminar en esa senda.[EN] In recent years the scientific community has significantly increased its use of virtual reality (VR) technologies in human behaviour research. In particular, the use of immersive VR has grown due to the introduction of affordable, high performance head mounted displays (HMDs). Among the fields that has strongly emerged in the last decade is affective computing, which combines psychophysiology, computer science, biomedical engineering and artificial intelligence in the development of systems that can automatically recognize emotions. The progress of affective computing is especially important in human behaviour research due to the central role that emotions play in many background processes, such as perception, decision-making, creativity, memory and social interaction. Several studies have tried to develop a reliable methodology to evoke and automatically identify emotional states using objective physiological measures and machine learning methods. However, the majority of previous studies used images, audio or video to elicit emotional statements; to the best of our knowledge, no previous research has developed an emotion recognition system using immersive VR. Although some previous studies analysed physiological responses in immersive VR, they did not use machine learning techniques for biosignal processing and classification. Moreover, a crucial concept when using VR for human behaviour research is validity: the capacity to evoke a response from the user in a simulated environment similar to the response that might be evoked in a physical environment. Although some previous studies have used psychological and cognitive dimensions to compare responses in real and virtual environments, few have extended this research to analyse physiological or behavioural responses. Moreover, to our knowledge, this is the first study to compare VR scenarios with their real-world equivalents using physiological measures coupled with machine learning algorithms, and to analyse the ability of VR to transfer and extrapolate insights obtained from VR environments to real environments. The main objective of this thesis is, using psycho-physiological and behavioural responses in combination with machine learning methods, and by performing a direct comparison between a real and virtual environment, to validate immersive VR as an emotion elicitation tool. To do so we develop an experimental protocol involving emotional 360º environments, an art exhibition in a real museum, and a highly-realistic 3D virtualization of the same art exhibition. This thesis provides novel contributions to the use of immersive VR in human behaviour research, particularly in relation to emotions. VR can help in the application of methodologies designed to present more realistic stimuli in the assessment of daily-life environments and situations, thus overcoming the current limitations of affective elicitation, which classically uses images, audio and video. Moreover, it analyses the validity of VR by performing a direct comparison using highly-realistic simulation. We believe that immersive VR will revolutionize laboratory-based emotion elicitation methods. Moreover, its synergy with physiological measurement and machine learning techniques will impact transversely in many other research areas, such as architecture, health, assessment, training, education, driving and marketing, and thus open new opportunities for the scientific community. The present dissertation aims to contribute to this progress.[CA] L'ús de la realitat virtual (RV) s'ha incrementat notablement en la comunitat científica per a la recerca del comportament humà. En particular, la RV immersiva ha crescut a causa de la democratització de les ulleres de realitat virtual o head mounted displays (HMD), que ofereixen un alt rendiment amb una reduïda inversió econòmica. Un dels camps que ha emergit amb força en l'última dècada és el Affective Computing, que combina psicofisiologia, informàtica, enginyeria biomèdica i intel·ligència artificial, desenvolupant sistemes que puguen reconéixer emocions automàticament. El seu progrés és especialment important en el camp de la recerca del comportament humà, a causa del paper fonamental que les emocions juguen en molts processos psicològics com la percepció, la presa de decisions, la creativitat, la memòria i la interacció social. Molts estudis s'han centrat en intentar obtenir una metodologia fiable per a evocar i automàticament identificar estats emocionals, utilitzant mesures fisiològiques objectives i mètodes d'aprenentatge automàtic. No obstant això, la major part dels estudis previs utilitzen imatges, àudios o vídeos per a generar els estats emocionals i, fins on arriba el nostre coneixement, cap d'ells ha desenvolupat un sistema de reconeixement emocional mitjançant l'ús de la RV immersiva. Encara que alguns treballs anteriors sí que analitzen les respostes fisiològiques en RV immersives, aquests no presenten models d'aprenentatge automàtic per a processament i classificació automàtica de biosenyals. A més, un concepte crucial quan s'utilitza la RV en la recerca del comportament humà és la validesa: la capacitat d'evocar respostes similars en un entorn virtual a les evocades per l'espai físic. Encara que alguns estudis previs han utilitzat dimensions psicològiques i cognitives per a comparar respostes entre entorns reals i virtuals, les recerques que analitzen respostes fisiològiques o comportamentals estan molt menys esteses. Segons els nostres coneixements, aquest és el primer treball que compara entorns físics amb la seua rèplica en RV, emprant respostes fisiològiques i algorismes d'aprenentatge automàtic i analitzant la capacitat de la RV de transferir i extrapolar les conclusions obtingudes a l'entorn real que s'està simulant. L'objectiu principal de la tesi és validar l'ús de la RV immersiva com una eina d'estimulació emocional usant respostes psicofisiològiques i comportamentals en combinació amb algorismes d'aprenentatge automàtic, així com realitzar una comparació directa entre un entorn real i virtual. Per a això, s'ha desenvolupat un protocol experimental que inclou entorns emocionals 360º, un museu real i una virtualització 3D altament realista del mateix museu. La tesi presenta noves contribucions de l'ús de la RV immersiva en la recerca del comportament humà, en particular quant a l'estudi de les emocions. Aquesta ajudarà a aplicar metodologies a estímuls més realistes per a avaluar entorns i situacions de la vida diària, superant les actuals limitacions de l'estimulació emocional que clàssicament ha inclòs imatges, àudios o vídeos. A més, en ella s'analitza la validesa de la RV realitzant una comparació directa usant una simulació altament realista. Creiem que la RV immersiva revolucionarà els mètodes d'estimulació emocional en entorns de laboratori. A més, la seua sinergia al costat de les mesures fisiològiques i les tècniques d'aprenentatge automàtic, impactaran transversalment en moltes àrees de recerca com l'arquitectura, la salut, l'avaluació psicològica, l'entrenament, l'educació, la conducció o el màrqueting, obrint un nou horitzó d'oportunitats per a la comunitat científica. La present tesi espera contribuir a caminar en aquesta senda.Marín Morales, J. (2020). Modelling human emotions using immersive virtual reality, physiological signals and behavioural responses [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/148717TESISCompendi

Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors

[EN] Affective Computing has emerged as an important field of study that aims to develop systems that can automatically recognize emotions. Up to the present, elicitation has been carried out with nonimmersive stimuli. This study, on the other hand, aims to develop an emotion recognition system for affective states evoked through Immersive Virtual Environments. Four alternative virtual rooms were designed to elicit four possible arousal-valence combinations, as described in each quadrant of the Circumplex Model of Affects. An experiment involving the recording of the electroencephalography (EEG) and electrocardiography (ECG) of sixty participants was carried out. A set of features was extracted from these signals using various state-of-the-art metrics that quantify brain and cardiovascular linear and nonlinear dynamics, which were input into a Support Vector Machine classifier to predict the subject's arousal and valence perception. The model's accuracy was 75.00% along the arousal dimension and 71.21% along the valence dimension. Our findings validate the use of Immersive Virtual Environments to elicit and automatically recognize different emotional states from neural and cardiac dynamics; this development could have novel applications in fields as diverse as Architecture, Health, Education and Videogames.This work was supported by the Ministerio de Economia y Competitividad. Spain (Project TIN2013-45736-R).Marín-Morales, J.; Higuera-Trujillo, JL.; Greco, A.; Guixeres Provinciale, J.; Llinares Millán, MDC.; Scilingo, EP.; Alcañiz Raya, ML.... (2018). Affective computing in virtual reality: emotion recognition from brain and heartbeat dynamics using wearable sensors. Scientific Reports. 8:1-15. https://doi.org/10.1038/s41598-018-32063-4S1158Picard, R. W. Affective computing. (MIT press, 1997).Picard, R. W. Affective Computing: Challenges. Int. J. Hum. Comput. Stud. 59, 55–64 (2003).Jerritta, S., Murugappan, M., Nagarajan, R. & Wan, K. Physiological signals based human emotion Recognition: a review. Signal Process. its Appl. (CSPA), 2011 IEEE 7th Int. Colloq. 410–415, https://doi.org/10.1109/CSPA.2011.5759912 (2011).Harms, M. B., Martin, A. & Wallace, G. L. Facial emotion recognition in autism spectrum disorders: A review of behavioral and neuroimaging studies. Neuropsychol. Rev. 20, 290–322 (2010).Koolagudi, S. G. & Rao, K. S. Emotion recognition from speech: A review. Int. J. Speech Technol. 15, 99–117 (2012).Gross, J. J. & Levenson, R. W. Emotion elicitation using films. Cogn. Emot. 9, 87–108 (1995).Lindal, P. J. & Hartig, T. Architectural variation, building height, and the restorative quality of urban residential streetscapes. J. Environ. Psychol. 33, 26–36 (2013).Ulrich, R. View through a window may influence recovery from surgery. Science (80-.). 224, 420–421 (1984).Fernández-Caballero, A. et al. Smart environment architecture for emotion detection and regulation. J. Biomed. Inform. 64, 55–73 (2016).Ekman, P. Basic Emotions. Handbook of cognition and emotion 45–60, https://doi.org/10.1017/S0140525X0800349X (1999).Posner, J., Russell, J. A. & Peterson, B. S. The circumplex model of affect: an integrative approach to affective neuroscience, cognitive development, and psychopathology. Dev. Psychopathol. 17, 715–34 (2005).Russell, J. A. & Mehrabian, A. Evidence for a three-factor theory of emotions. J. Res. Pers. 11, 273–294 (1977).Calvo, R. A. & D’Mello, S. Affect detection: An interdisciplinary review of models, methods, and their applications. IEEE Trans. Affect. Comput. 1, 18–37 (2010).Valenza, G. et al. Combining electroencephalographic activity and instantaneous heart rate for assessing brain–heart dynamics during visual emotional elicitation in healthy subjects. Philos. Trans. R. Soc. A Math. Phys. Eng. Sci. 374, 20150176 (2016).Valenza, G., Lanata, A. & Scilingo, E. P. The role of nonlinear dynamics in affective valence and arousal recognition. IEEE Trans. Affect. Comput. 3, 237–249 (2012).Valenza, G., Citi, L., Lanatá, A., Scilingo, E. P. & Barbieri, R. Revealing real-time emotional responses: a personalized assessment based on heartbeat dynamics. Sci. Rep. 4, 4998 (2014).Valenza, G. et al. Wearable monitoring for mood recognition in bipolar disorder based on history-dependent long-term heart rate variability analysis. IEEE J. Biomed. Heal. Informatics 18, 1625–1635 (2014).Piwek, L., Ellis, D. A., Andrews, S. & Joinson, A. The Rise of Consumer Health Wearables: Promises and Barriers. PLoS Med. 13, 1–9 (2016).Xu, J., Mitra, S., Van Hoof, C., Yazicioglu, R. & Makinwa, K. A. A. Active Electrodes for Wearable EEG Acquisition: Review and Electronics Design Methodology. IEEE Rev. Biomed. Eng. 3333, 1–1 (2017).Kumari, P., Mathew, L. & Syal, P. Increasing trend of wearables and multimodal interface for human activity monitoring: A review. Biosens. Bioelectron. 90, 298–307 (2017).He, C., Yao, Y. & Ye, X. An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors. In Wearable Sensors and Robots: Proceedings of International Conference on Wearable Sensors and Robots 2015 (eds Yang, C., Virk, G. S. & Yang, H.) 15–25. https://doi.org/10.1007/978-981-10-2404-7_2 (Springer Singapore, 2017).Nakisa, B., Rastgoo, M. N., Tjondronegoro, D. & Chandran, V. Evolutionary computation algorithms for feature selection of EEG-based emotion recognition using mobile sensors. Expert Syst. Appl. 93, 143–155 (2018).Kory Jacqueline, D. & Sidney, K. Affect Elicitation for Affective Computing. In The Oxford Handbook of Affective Computing 371–383 (2014).Ekman, P. The directed facial action task. In Handbook of emotion elicitation and assessment 47–53 (2007).Harmon-Jones, E., Amodio, D. M. & Zinner, L. R. Social psychological methods of emotion elicitation. Handb. Emot. elicitation Assess. 91–105, https://doi.org/10.2224/sbp.2007.35.7.863 (2007)Roberts, N. A., Tsai, J. L. & Coan, J. A. Emotion elicitation using dyadic interaction task. Handbook of Emotion Elicitation and Assessment 106–123 (2007).Nardelli, M., Valenza, G., Greco, A., Lanata, A. & Scilingo, E. P. Recognizing emotions induced by affective sounds through heart rate variability. IEEE Trans. Affect. Comput. 6, 385–394 (2015).Kim, J. Emotion Recognition Using Speech and Physiological Changes. Robust Speech Recognit. Underst. 265–280 (2007).Soleymani, M., Pantic, M. & Pun, T. Multimodal emotion recognition in response to videos (Extended abstract). 2015 Int. Conf. Affect. Comput. Intell. Interact. ACII 2015 3, 491–497 (2015).Baños, R. M. et al. Immersion and Emotion: Their Impact on the Sense of Presence. CyberPsychology Behav. 7, 734–741 (2004).Giglioli, I. A. C., Pravettoni, G., Martín, D. L. S., Parra, E. & Raya, M. A. A novel integrating virtual reality approach for the assessment of the attachment behavioral system. Front. Psychol. 8, 1–7 (2017).Marín-Morales, J., Torrecilla, C., Guixeres, J. & Llinares, C. Methodological bases for a new platform for the measurement of human behaviour in virtual environments. DYNA 92, 34–38 (2017).Vince, J. Introduction to virtual reality. (Media, Springer Science & Business, 2004).Alcañiz, M., Baños, R., Botella, C. & Rey, B. The EMMA Project: Emotions as a Determinant of Presence. PsychNology J. 1, 141–150 (2003).Vecchiato, G. et al. Neurophysiological correlates of embodiment and motivational factors during the perception of virtual architectural environments. Cogn. Process. 16, 425–429 (2015).Slater, M. & Wilbur, S. A Framework for Immersive Virtual Environments (FIVE): Speculations on the Role of Presence in Virtual Environments. Presence Teleoperators Virtual Environ. 6, 603–616 (1997).Riva, G. et al. Affective Interactions Using Virtual Reality: The Link between Presence and Emotions. CyberPsychology Behav. 10, 45–56 (2007).Baños, R. M. et al Changing induced moods via virtual reality. In International Conference on Persuasive Technology (ed. Springer, Berlin, H.) 7–15, https://doi.org/10.1007/11755494_3 (2006).Baños, R. M. et al. Positive mood induction procedures for virtual environments designed for elderly people. Interact. Comput. 24, 131–138 (2012).Gorini, A. et al. Emotional Response to Virtual Reality Exposure across Different Cultures: The Role of the AttributionProcess. CyberPsychology Behav. 12, 699–705 (2009).Gorini, A., Capideville, C. S., De Leo, G., Mantovani, F. & Riva, G. The Role of Immersion and Narrative in Mediated Presence: The Virtual Hospital Experience. Cyberpsychology, Behav. Soc. Netw. 14, 99–105 (2011).Chirico, A. et al. Effectiveness of Immersive Videos in Inducing Awe: An Experimental Study. Sci. Rep. 7, 1–11 (2017).Blascovich, J. et al. Immersive Virtual Environment Technology as a Methodological Tool for Social Psychology. Psychol. Inq. 7965, 103–124 (2012).Peperkorn, H. M., Alpers, G. W. & Mühlberger, A. Triggers of fear: Perceptual cues versus conceptual information in spider phobia. J. Clin. Psychol. 70, 704–714 (2014).McCall, C., Hildebrandt, L. K., Bornemann, B. & Singer, T. Physiophenomenology in retrospect: Memory reliably reflects physiological arousal during a prior threatening experience. Conscious. Cogn. 38, 60–70 (2015).Hildebrandt, L. K., Mccall, C., Engen, H. G. & Singer, T. Cognitive flexibility, heart rate variability, and resilience predict fine-grained regulation of arousal during prolonged threat. Psychophysiology 53, 880–890 (2016).Notzon, S. et al. Psychophysiological effects of an iTBS modulated virtual reality challenge including participants with spider phobia. Biol. Psychol. 112, 66–76 (2015).Amaral, C. P., Simões, M. A., Mouga, S., Andrade, J. & Castelo-Branco, M. A novel Brain Computer Interface for classification of social joint attention in autism and comparison of 3 experimental setups: A feasibility study. J. Neurosci. Methods 290, 105–115 (2017).Eudave, L. & Valencia, M. Physiological response while driving in an immersive virtual environment. 2017 IEEE 14th Int. Conf. Wearable Implant. Body Sens. Networks 145–148, https://doi.org/10.1109/BSN.2017.7936028 (2017).Sharma, G. et al. Influence of landmarks on wayfinding and brain connectivity in immersive virtual reality environment. Front. Psychol. 8, 1–12 (2017).Bian, Y. et al. A framework for physiological indicators of flow in VR games: construction and preliminary evaluation. Pers. Ubiquitous Comput. 20, 821–832 (2016).Egan, D. et al. An evaluation of Heart Rate and Electrodermal Activity as an Objective QoE Evaluation method for Immersive Virtual Reality Environments. 3–8, https://doi.org/10.1109/QoMEX.2016.7498964 (2016).Meehan, M., Razzaque, S., Insko, B., Whitton, M. & Brooks, F. P. Review of four studies on the use of physiological reaction as a measure of presence in stressful virtual environments. Appl. Psychophysiol. Biofeedback 30, 239–258 (2005).Higuera-Trujillo, J. L., López-Tarruella Maldonado, J. & Llinares Millán, C. Psychological and physiological human responses to simulated and real environments: A comparison between Photographs, 360° Panoramas, and Virtual Reality. Appl. Ergon. 65, 398–409 (2016).Felnhofer, A. et al. Is virtual reality emotionally arousing? Investigating five emotion inducing virtual park scenarios. Int. J. Hum. Comput. Stud. 82, 48–56 (2015).Anderson, A. P. et al. Relaxation with Immersive Natural Scenes Presented Using Virtual Reality. Aerosp. Med. Hum. Perform. 88, 520–526 (2017).Higuera, J. L. et al. Emotional cartography in design: A novel technique to represent emotional states altered by spaces. In D and E 2016: 10th International Conference on Design and Emotion 561–566 (2016).Kroenke, K., Spitzer, R. L. & Williams, J. B. W. The PHQ-9: Validity of a brief depression severity measure. J. Gen. Intern. Med. 16, 606–613 (2001).Bradley, M. M. & Lang, P. J. Measuring emotion: The self-assessment manikin and the semantic differential. J. Behav. Ther. Exp. Psychiatry 25, 49–59 (1994).Lang, P. J., Bradley, M. M. & Cuthbert, B. N. International Affective Picture System (IAPS): Technical Manual and Affective Ratings. NIMH Cent. Study Emot. Atten. 39–58, https://doi.org/10.1027/0269-8803/a000147 (1997).Nanda, U., Pati, D., Ghamari, H. & Bajema, R. Lessons from neuroscience: form follows function, emotions follow form. Intell. Build. Int. 5, 61–78 (2013).Russell, J. A. A circumplex model of affect. J. Pers. Soc. Psychol. 39, 1161–1178 (1980).Sejima, K. Kazuyo Sejima. 1988–1996. El Croquis 15 (1996).Ochiai, H. et al. Physiological and Psychological Effects of Forest Therapy on Middle-Aged Males with High-NormalBlood Pressure. Int. J. Environ. Res. Public Health 12, 2532–2542 (2015).Noguchi, H. & Sakaguchi, T. Effect of illuminance and color temperature on lowering of physiological activity. Appl. Hum. Sci. 18, 117–123 (1999).Küller, R., Mikellides, B. & Janssens, J. Color, arousal, and performance—A comparison of three experiments. Color Res. Appl. 34, 141–152 (2009).Yildirim, K., Hidayetoglu, M. L. & Capanoglu, A. Effects of interior colors on mood and preference: comparisons of two living rooms. Percept. Mot. Skills 112, 509–524 (2011).Hogg, J., Goodman, S., Porter, T., Mikellides, B. & Preddy, D. E. Dimensions and determinants of judgements of colour samples and a simulated interior space by architects and non‐architects. Br. J. Psychol. 70, 231–242 (1979).Jalil, N. A., Yunus, R. M. & Said, N. S. Environmental Colour Impact upon Human Behaviour: A Review. Procedia - Soc. Behav. Sci. 35, 54–62 (2012).Jacobs, K. W. & Hustmyer, F. E. Effects of four psychological primary colors on GSR, heart rate and respiration rate. Percept. Mot. Skills 38, 763–766 (1974).Jin, H. R., Yu, M., Kim, D. W., Kim, N. G. & Chung, A. S. W. Study on Physiological Responses to Color Stimulation. In International Association of Societies of Design Research (ed. Poggenpohl, S.) 1969–1979 (Korean Society of Design Science, 2009).Vartanian, O. et al. Impact of contour on aesthetic judgments and approach-avoidance decisions in architecture. Proc. Natl. Acad. Sci. 110, 1–8 (2013).Tsunetsugu, Y., Miyazaki, Y. & Sato, H. Visual effects of interior design in actual-size living rooms on physiological responses. Build. Environ. 40, 1341–1346 (2005).Stamps, A. E. Physical Determinants of Preferences for Residential Facades. Environ. Behav. 31, 723–751 (1999).Berlyne, D. E. Novelty, Complexity, and Hedonic Value. Percept. Psychophys. 8, 279–286 (1970).Krueger, R. A. & Casey, M. Focus groups: a practical guide for applied research. (Sage Publications, 2000).Acharya, U. R., Joseph, K. P., Kannathal, N., Lim, C. M. & Suri, J. S. Heart rate variability: A review. Med. Biol. Eng. Comput. 44, 1031–1051 (2006).Tarvainen, M. P., Niskanen, J. P., Lipponen, J. A., Ranta-aho, P. O. & Karjalainen, P. A. Kubios HRV - Heart rate variability analysis software. Comput. Methods Programs Biomed. 113, 210–220 (2014).Pan, J. & Tompkins, W. J. A real-time QRS detection algorithm. Biomed. Eng. IEEE Trans. 1, 230–236 (1985).Tarvainen, M. P., Ranta-aho, P. O. & Karjalainen, P. A. An advanced detrending method with application to HRV analysis. IEEE Trans. Biomed. Eng. 49, 172–175 (2002).Valenza, G. et al. Predicting Mood Changes in Bipolar Disorder Through HeartbeatNonlinear Dynamics. IEEE J. Biomed. Heal. Informatics 20, 1034–1043 (2016).Pincus, S. & Viscarello, R. Approximate Entropy A regularity measure for fetal heart rate analysis. Obstet. Gynecol. 79, 249–255 (1992).Richman, J. & Moorman, J. Physiological time-series analysis using approximate entropy and sample entropy. Am J Physiol Hear. Circ Physiol 278, H2039–H2049 (2000).Peng, C.-K., Havlin, S., Stanley, H. E. & Goldberger, A. L. Quantification of scaling exponents and crossover phenomena in nonstationary heartbeat time series. Chaos 5, 82–87 (1995).Grassberger, P. & Procaccia, I. Characterization of strange attractors. Phys. Rev. Lett. 50, 346–349 (1983).Delorme, A. & Makeig, S. EEGLAB: An open source toolbox for analysis of single-trial EEG dynamics including independent component analysis. J. Neurosci. Methods 134, 9–21 (2004).Colomer Granero, A. et al. A Comparison of Physiological Signal Analysis Techniques and Classifiers for Automatic Emotional Evaluation of Audiovisual Contents. Front. Comput. Neurosci. 10, 1–14 (2016).Kober, S. E., Kurzmann, J. & Neuper, C. Cortical correlate of spatial presence in 2D and 3D interactive virtual reality: An EEG study. Int. J. Psychophysiol. 83, 365–374 (2012).Hyvärinen, A. & Oja, E. Independent component analysis: Algorithms and applications. Neural Networks 13, 411–430 (2000).Welch, P. D. The Use of Fast Fourier Transform for the Estimation of Power Spectra: A Method Based on Time Aver. aging Over Short, Modified Periodograms. IEEE Trans. AUDIO Electroacoust. 15, 70–73 (1967).Mormann, F., Lehnertz, K., David, P. & Elger, E. C. Mean phase coherence as a measure for phase synchronization and its application to the EEG of epilepsy patients. Phys. D Nonlinear Phenom. 144, 358–369 (2000).Jolliffe, I. T. Principal Component Analysis, Second Edition. Encycl. Stat. Behav. Sci. 30, 487 (2002).Schöllkopf, B., Smola, A. J., Williamson, R. C. & Bartlett, P. L. New support vector algorithms. Neural Comput 12, 1207–1245 (2000).Yan, K. & Zhang, D. Feature selection and analysis on correlated gas sensor data with recursive feature elimination. Sensors Actuators, B Chem. 212, 353–363 (2015).Chang, C.-C. & Lin, C.-J. Libsvm: A Library for Support Vector Machines. ACM Trans. Intell. Syst. Technol. 2, 1–27 (2011).Lewis, P. A., Critchley, H. D., Rotshtein, P. & Dolan, R. J. Neural correlates of processing valence and arousal in affective words. Cereb. Cortex 17, 742–748 (2007).McCall, C., Hildebrandt, L. K., Hartmann, R., Baczkowski, B. M. & Singer, T. Introducing the Wunderkammer as a tool for emotion research: Unconstrained gaze and movement patterns in three emotionally evocative virtual worlds. Comput. Human Behav. 59, 93–107 (2016).Blake, J. & Gurocak, H. B. Haptic glove with MR brakes for virtual reality. IEEE/ASME Trans. Mechatronics 14, 606–615 (2009).Heydarian, A. et al. Immersive virtual environments versus physical built environments: A benchmarking study for building design and user-built environment explorations. Autom. Constr. 54, 116–126 (2015).Kuliga, S. F., Thrash, T., Dalton, R. C. & Hölscher, C. Virtual reality as an empirical research tool - Exploring user experience in a real building and a corresponding virtual model. Comput. Environ. Urban Syst. 54, 363–375 (2015).Yeom, D., Choi, J.-H. & Zhu, Y. Investigation of the Physiological Differences between Immersive Virtual Environment and Indoor Enviorment in a Building. Indoor adn Built Enviornment 0, Accept (2017).Combrisson, E. & Jerbi, K. Exceeding chance level by chance: The caveat of theoretical chance levels in brain signal classification and statistical assessment of decoding accuracy. J. Neurosci. Methods 250, 126–136 (2015).He, C., Yao, Y. & Ye, X. An Emotion Recognition System Based on Physiological Signals Obtained by Wearable Sensors. In Wearable Sensors and Robots: Proceedings of International Conference on Wearable Sensors and Robots 2015 (eds. Yang, C., Virk, G. S. & Yang, H.) 15–25, https://doi.org/10.1007/978-981-10-2404-7_2 (Springer Singapore, 2017)

Frontal EEG Asymmetry and Middle Line Power Difference in Discrete Emotions

A traditional model of emotion cannot explain the differences in brain activities between two discrete emotions that are similar in the valence-arousal coordinate space. The current study elicited two positive emotions (amusement and tenderness) and two negative emotions (anger and fear) that are similar in both valence and arousal dimensions to examine the differences in brain activities in these emotional states. Frontal electroencephalographic (EEG) asymmetry and midline power in three bands (theta, alpha and beta) were measured when participants watched affective film excerpts. Significant differences were detected between tenderness and amusement on FP1/FP2 theta asymmetry, F3/F4 theta and alpha asymmetry. Significant differences between anger and fear on FP1/FP2 theta asymmetry and F3/F4 alpha asymmetry were also observed. For midline power, midline theta power could distinguish two negative emotions, while midline alpha and beta power could effectively differentiate two positive emotions. Liking and dominance were also related to EEG features. Stepwise multiple linear regression results revealed that frontal alpha and theta asymmetry could predict the subjective feelings of two positive and two negative emotions in different patterns. The binary classification accuracy, which used EEG frontal asymmetry and midline power as features and support vector machine (SVM) as classifiers, was as high as 64.52% for tenderness and amusement and 78.79% for anger and fear. The classification accuracy was improved after adding these features to other features extracted across the scalp. These findings indicate that frontal EEG asymmetry and midline power might have the potential to recognize discrete emotions that are similar in the valence-arousal coordinate space

The transmission of music into the human uterus and the response to music of the human fetus and neonate

The aim of this study was to investigate whether music influences human life before birth. In order to determine the existence and character of music in the uterine acoustic environment, a study was conducted involving the insertion of a hydrophone through the cervix, next to the fetal head. The investigation was conducted on eight women in early labour. The average residual uterine sound of the eight subjects was measured at 65 dBA (A-weighted) re 20 µ.Pa in a 1 O KHz band, RMS averaged over 32-second records. Above this emerged the maternal voice, an external female voice and a male voice presented at approximately 65 dB (linear weighted). Pure tones between 50 Hz and 1 O KHz and orchestral music, all presented at 80 dB (linear weighted), were also shown to emerge above the residual uterine sound. Attenuation of external sound was observed to vary as a function of frequency, with less attenuation of lower frequencies. It was determined that the music was transmitted into the uterus without sufficient distortion to significantly alter the recognisable characteristics of the music. The fetal heart rate (FHA) response to a music stimulus (MS) and a vibroacoustic stimulus (VS) was measured in 40 subjects. Gestational age of the fetuses ranged from 32 to 42 weeks. The study included a control period with no acoustic stimulation; a period with the presentation of 5 music stimuli; and a period with the presentation of 5 vibroacoustic stimuli. A change in the FHA of 15 beats per minute or greater, lasting 15 seconds and occurring within 15 seconds of at least 2 of the 5 stimuli (or a tachycardia of greater than 15 beats per minute above the resting baseline, sustained for one minute or longer) was considered to be a positive response. The MS elicited a positive response in 35 of the fetuses (the 5 non-responses occurring in a period of low FHA variability) and all 40 fetuses responded to the VS (regardless of arousal state). In the third study, mothers attending childbirth education classes volunteered to listen to a prescribed music excerpt twice daily from the 34th week of pregnancy. Ten neonates (all clinically normal) were tested betw~en the 2nd and 5th day after birth. Investigators observed the effect of two music sti:Tiuli, the prescribed stimulus and a non-prescribed stimulus, on neonatal sucking of a non-nutritive nipple. A five-minute control period with no stimulation was compared with a ten-minute period during which two music stimuli were presented. By random allocation, either the prescribed music stimulus (PM) or the nonprescribed music (NM) was presented contingent upon sucking pressure. If a sucking burst was initiated, the PM stimulus was activated. On cessation of sucking, the NM stimulus was activated. Randomly, the procedure would be reversed for some of the subjects, where initiation of sucking activated the NM stimulus and cessation of sucking activated the PM stimulus. It was determined that the inter-burst intervals during the music period were significantly extended when coinciding with the PM stimulus and significantly shortened when coinciding with the NM stimulus.The studies indicated that music is transmitted into the uterus with insufficient distortion to alter the character of the music; that the normal fetus responds to a music stimulus from at least the 32nd week of gestation; and that the neonate alters the normal sucking pattern to activate longer periods of a music stimulus which has been repeatedly presented during the intrauterine stage and shorter periods of a novel music stimulus