Machine Learning Methods for functional Near Infrared Spectroscopy

Identification of user state is of interest in a wide range of disciplines that fall under the umbrella of human machine interaction. Functional Near Infra-Red Spectroscopy (fNIRS) device is a relatively new device that enables inference of brain activity through non-invasively pulsing infra-red light into the brain. The fNIRS device is particularly useful as it has a better spatial resolution than the Electroencephalograph (EEG) device that is most commonly used in Human Computer Interaction studies under ecologically valid settings. But this key advantage of fNIRS device is underutilized in current literature in the fNIRS domain. We propose machine learning methods that capture this spatial nature of the human brain activity using a novel preprocessing method that uses `Region of Interest\u27 based feature extraction. Experiments show that this method outperforms the F1 score achieved previously in classifying `low\u27 vs `high\u27 valence state of a user. We further our analysis by applying a Convolutional Neural Network (CNN) to the fNIRS data, thus preserving the spatial structure of the data and treating the data similar to a series of images to be classified. Going further, we use a combination of CNN and Long Short-Term Memory (LSTM) to capture the spatial and temporal behavior of the fNIRS data, thus treating it similar to a video classification problem. We show that this method improves upon the accuracy previously obtained by valence classification methods using EEG or fNIRS devices. Finally, we apply the above model to a problem in classifying combined task-load and performance in an across-subject, across-task scenario of a Human Machine Teaming environment in order to achieve optimal productivity of the system

Speech Processes for Brain-Computer Interfaces

Speech interfaces have become widely used and are integrated in many applications and devices. However, speech interfaces require the user to produce intelligible speech, which might be hindered by loud environments, concern to bother bystanders or the general in- ability to produce speech due to disabilities. Decoding a usera s imagined speech instead of actual speech would solve this problem. Such a Brain-Computer Interface (BCI) based on imagined speech would enable fast and natural communication without the need to actually speak out loud. These interfaces could provide a voice to otherwise mute people. This dissertation investigates BCIs based on speech processes using functional Near In- frared Spectroscopy (fNIRS) and Electrocorticography (ECoG), two brain activity imaging modalities on opposing ends of an invasiveness scale. Brain activity data have low signal- to-noise ratio and complex spatio-temporal and spectral coherence. To analyze these data, techniques from the areas of machine learning, neuroscience and Automatic Speech Recog- nition are combined in this dissertation to facilitate robust classification of detailed speech processes while simultaneously illustrating the underlying neural processes. fNIRS is an imaging modality based on cerebral blood flow. It only requires affordable hardware and can be set up within minutes in a day-to-day environment. Therefore, it is ideally suited for convenient user interfaces. However, the hemodynamic processes measured by fNIRS are slow in nature and the technology therefore offers poor temporal resolution. We investigate speech in fNIRS and demonstrate classification of speech processes for BCIs based on fNIRS. ECoG provides ideal signal properties by invasively measuring electrical potentials artifact- free directly on the brain surface. High spatial resolution and temporal resolution down to millisecond sampling provide localized information with accurate enough timing to capture the fast process underlying speech production. This dissertation presents the Brain-to- Text system, which harnesses automatic speech recognition technology to decode a textual representation of continuous speech from ECoG. This could allow to compose messages or to issue commands through a BCI. While the decoding of a textual representation is unparalleled for device control and typing, direct communication is even more natural if the full expressive power of speech - including emphasis and prosody - could be provided. For this purpose, a second system is presented, which directly synthesizes neural signals into audible speech, which could enable conversation with friends and family through a BCI. Up to now, both systems, the Brain-to-Text and synthesis system are operating on audibly produced speech. To bridge the gap to the final frontier of neural prostheses based on imagined speech processes, we investigate the differences between audibly produced and imagined speech and present first results towards BCI from imagined speech processes. This dissertation demonstrates the usage of speech processes as a paradigm for BCI for the first time. Speech processes offer a fast and natural interaction paradigm which will help patients and healthy users alike to communicate with computers and with friends and family efficiently through BCIs

Social and Affective Neuroscience of Everyday Human Interaction

This Open Access book presents the current state of the art knowledge on social and affective neuroscience based on empirical findings. This volume is divided into several sections first guiding the reader through important theoretical topics within affective neuroscience, social neuroscience and moral emotions, and clinical neuroscience. Each chapter addresses everyday social interactions and various aspects of social interactions from a different angle taking the reader on a diverse journey. The last section of the book is of methodological nature. Basic information is presented for the reader to learn about common methodologies used in neuroscience alongside advanced input to deepen the understanding and usability of these methods in social and affective neuroscience for more experienced readers

2023 SOARS Conference Program

Program for the 2023 Showcase of Osprey Advancements in Research and Scholarship (SOARS

Social and Affective Neuroscience of Everyday Human Interaction

This Open Access book presents the current state of the art knowledge on social and affective neuroscience based on empirical findings. This volume is divided into several sections first guiding the reader through important theoretical topics within affective neuroscience, social neuroscience and moral emotions, and clinical neuroscience. Each chapter addresses everyday social interactions and various aspects of social interactions from a different angle taking the reader on a diverse journey. The last section of the book is of methodological nature. Basic information is presented for the reader to learn about common methodologies used in neuroscience alongside advanced input to deepen the understanding and usability of these methods in social and affective neuroscience for more experienced readers

Laterality in pigs and its links with personality, emotions and animal welfare

The study of laterality (i.e. asymmetries of brain and behaviour) is a potential non-invasive approach to gain insights into the common neural mechanisms underpinning both personality and emotions in animals. The hypotheses underlying this thesis state that the left (respectively the right) hemisphere regulates approach or positive (respectively avoidance or negative) emotions. The goal of this thesis was to study lateralized motor functions (Study 1) and their associations with personality indices (Study 2), and the effect of monocular viewing on emotional reactions (Study 3) in pigs.Beobachtungsstudien zur Lateralität (d.h. Asymmetrien von Gehirn und Verhalten) sind ein potenzieller nicht-invasiver Ansatz, um die gemeinsamen neuronalen Grundlagen von Persönlichkeit und Emotion aufzuklären. Gemäß den dieser Dissertation zugrundeliegenden Hypothesen steuert die linke (bzw. die rechte) Gehirnhälfte Annäherungs- oder positive (bzw. Rückzugs- oder negative) Emotionen. Das Ziel war, sowohl lateralisierte motorische Funktionen und ihre Interaktionen mit Persönlichkeitsindizes, als auch die Wirkung monokularen Sehens auf emotionale Reaktionen bei Hausschweinen zu untersuchen

Brain Music : Sistema generativo para la creación de música simbólica a partir de respuestas neuronales afectivas

gráficas, tablasEsta tesis de maestría presenta una metodología de aprendizaje profundo multimodal innovadora que fusiona un modelo de clasificación de emociones con un generador musical, con el propósito de crear música a partir de señales de electroencefalografía, profundizando así en la interconexión entre emociones y música. Los resultados alcanzan tres objetivos específicos: Primero, ya que el rendimiento de los sistemas interfaz cerebro-computadora varía considerablemente entre diferentes sujetos, se introduce un enfoque basado en la transferencia de conocimiento entre sujetos para mejorar el rendimiento de individuos con dificultades en sistemas de interfaz cerebro-computadora basados en el paradigma de imaginación motora. Este enfoque combina datos de EEG etiquetados con datos estructurados, como cuestionarios psicológicos, mediante un método de "Kernel Matching CKA". Utilizamos una red neuronal profunda (Deep&Wide) para la clasificación de la imaginación motora. Los resultados destacan su potencial para mejorar las habilidades motoras en interfaces cerebro-computadora. Segundo, proponemos una técnica innovadora llamada "Labeled Correlation Alignment"(LCA) para sonificar respuestas neurales a estímulos representados en datos no estructurados, como música afectiva. Esto genera características musicales basadas en la actividad cerebral inducida por las emociones. LCA aborda la variabilidad entre sujetos y dentro de sujetos mediante el análisis de correlación, lo que permite la creación de envolventes acústicos y la distinción entre diferente información sonora. Esto convierte a LCA en una herramienta prometedora para interpretar la actividad neuronal y su reacción a estímulos auditivos. Finalmente, en otro capítulo, desarrollamos una metodología de aprendizaje profundo de extremo a extremo para generar contenido musical MIDI (datos simbólicos) a partir de señales de actividad cerebral inducidas por música con etiquetas afectivas. Esta metodología abarca el preprocesamiento de datos, el entrenamiento de modelos de extracción de características y un proceso de emparejamiento de características mediante Deep Centered Kernel Alignment, lo que permite la generación de música a partir de señales EEG. En conjunto, estos logros representan avances significativos en la comprensión de la relación entre emociones y música, así como en la aplicación de la inteligencia artificial en la generación musical a partir de señales cerebrales. Ofrecen nuevas perspectivas y herramientas para la creación musical y la investigación en neurociencia emocional. Para llevar a cabo nuestros experimentos, utilizamos bases de datos públicas como GigaScience, Affective Music Listening y Deap Dataset (Texto tomado de la fuente)This master’s thesis presents an innovative multimodal deep learning methodology that combines an emotion classification model with a music generator, aimed at creating music from electroencephalography (EEG) signals, thus delving into the interplay between emotions and music. The results achieve three specific objectives: First, since the performance of brain-computer interface systems varies significantly among different subjects, an approach based on knowledge transfer among subjects is introduced to enhance the performance of individuals facing challenges in motor imagery-based brain-computer interface systems. This approach combines labeled EEG data with structured information, such as psychological questionnaires, through a "Kernel Matching CKA"method. We employ a deep neural network (Deep&Wide) for motor imagery classification. The results underscore its potential to enhance motor skills in brain-computer interfaces. Second, we propose an innovative technique called "Labeled Correlation Alignment"(LCA) to sonify neural responses to stimuli represented in unstructured data, such as affective music. This generates musical features based on emotion-induced brain activity. LCA addresses variability among subjects and within subjects through correlation analysis, enabling the creation of acoustic envelopes and the distinction of different sound information. This makes LCA a promising tool for interpreting neural activity and its response to auditory stimuli. Finally, in another chapter, we develop an end-to-end deep learning methodology for generating MIDI music content (symbolic data) from EEG signals induced by affectively labeled music. This methodology encompasses data preprocessing, feature extraction model training, and a feature matching process using Deep Centered Kernel Alignment, enabling music generation from EEG signals. Together, these achievements represent significant advances in understanding the relationship between emotions and music, as well as in the application of artificial intelligence in musical generation from brain signals. They offer new perspectives and tools for musical creation and research in emotional neuroscience. To conduct our experiments, we utilized public databases such as GigaScience, Affective Music Listening and Deap DatasetMaestríaMagíster en Ingeniería - Automatización IndustrialInvestigación en Aprendizaje Profundo y señales BiológicasEléctrica, Electrónica, Automatización Y Telecomunicaciones.Sede Manizale

Neuromarketing and Consumer Neuroscience – The Evolution and Current State of the Art, an Integrative Review

Abstract Neuromarketing or consumer neuroscience is a rather new field of research, which utilizes neuroscientific tools, such as EEG (Electroencephalography) and fMRI (functional Magnetic Resonance Imaging) to identify marketing related phenomena, such as decision making, liking, brand image etc. This Master Thesis takes an Integrative Review approach, combining narrative review, historical review and systematic review frameworks for researching neuromarketing overall to create a timeline of the evolution of the field during it’s’ still rather short existence. 42 articles from the years 2002-2019 were qualitatively analysed to obtain this overall image. The research is of qualitative nature and focuses on identifying different themes surrounding neuromarketing and consumer neuroscience and the goal is to create an overall image of the field to provide valuable information for scholars as well as managers on how to implement neuromarketing in future research and in practice. Three key areas/themes were identified where the evolution of neuromarketing and consumer neuroscience can be best observed. These key areas were 1. The understanding of quality 2. The evolution of attitudes and 3. The growing interest towards the field vs. lowering entry barriers. In these almost twenty years, the understanding of what is good quality neuromarketing research has evolved from almost inexistent to a solid understanding, while the amount of actual quality neuromarketing research is still to follow. The attitudes started from fairy-tale-like high hopes, like finding the “holy grail” of marketing, which would be a “buy button” inside the consumer’s head and painting dystopian future scenarios, where the consumer is stripped down from his autonomy and the marketer’s control their minds. Since then attitudes have evolved to a much more realistic optimism, where the potential is recognized but not overhyped and the dystopian imagery has shifted to reasonable ethical concerns, like neuroimaging safety. As a characteristic for the field, the entry barriers for new researchers have been rather high. As time has went on, the interest towards the field has simultaneously grown as a lot of valuable research work has been done to considerably lower the entry barriers. In the light of these results the author suggests that managers should remain cautious should they invest on neuromarketing consultancy or engage in decision making based on neuromarketing findings. This is mainly, because the current literature consists mostly of conceptual research articles and the current empirical findings are largely of subpar quality and scattered in nature. Additionally, there are many consultancy companies offering neuromarketing based consultancy, which should consequently be approached with caution. For the scholars, neuromarketing remains an interesting field of study and it seems that the current understanding offers a solid groundwork for quality future research. The author suggests that future research would start focusing on empirical research, since the necessary theoretical foundation is solid and offers sufficient guidance to conduct quality empirical research. Increasing the amount of empirical research would also seem like an organic evolutionary step at the current stage of neuromarketing as a research field. New solid empirical findings would finally claim neuromarketing’s long recognized potential. Tiivistelmä Neuromarkkinointi (neuromarketing) tai Kuluttajan aivotutkimus (consumer neuroscience) on melko uusi tieteenala, joka tutkii markkinoinnillisia ilmiöitä, kuten päätöksentekoa, mieltymyksiä (liking), brändäystä, imagoa jne. aivotutkimuksellisia keinoja, kuten EEG (aivosähkökäyrä) tai fMRI (magneettikuvaus) apunaan käyttäen. Tämä Maisterin tutkielma ottaa yhdistelevän katsauksen (integrative review) näkökulman, yhdistellen kertomuksellista, systemaattista ja historiallisen katsauksen viitekehyksiä, muodostamaan aikajanan neuromarkkinoinnin kehityksestä tieteenalana. Tutkielma on kvalitatiivinen ja tarkoituksena on tunnistaa neuromarkkinoinnin tieteenalan ympärillä kunakin aikana vallitsevia teemoja ja tätä kautta muodostaa kokonaiskuva tieteenalasta ja sen kehityksestä. Tätä varten kirjoittaja analysoi kvalitatiivisesti 42 artikkelia vuosilta 2002–2019. Kokonaiskuvan muodostamisen tarkoituksena on tarjota informaatiota johtajille, sekä tutkijoille, jotta he voisivat tehdä hyviä päätöksiä koskien neuromarkkinointia esimerkiksi investoinneissa tai tulevassa tutkimuksessa. Tutkimuksessa tunnistettiin kolme pääteemaa, joista neuromarkkinoinnin evoluutiota on mielekkäintä tarkkailla. Nämä pääkohdat olivat 1. Laadun ymmärrys neuromarkkinointitutkimuksessa 2. Asenteiden kehitys tieteenalaa kohtaan ja 3. Kasvava kiinnostus tieteenalaa kohtaan vs. alalle sisääntulon kynnysten alentaminen. Näiden melkein kahden vuosikymmenen aikana, ymmärrys laadukkaasta neuromarkkinointitutkimuksesta kehittyi lähes olemattomasta vakaaseen ymmärrykseen. Laadukkaan tutkimuksen määrän kasvun odotetaan seuraavan perässä. Asenteet puolestaan alkoivat satumaisen korkeista odotuksista, kuten markkinoinnin “Graalin maljan” eli “Ostonapin” mahdollisesta löytämisestä kuluttajan aivoista ja dystopian kaltaisten tulevaisuuskuvien maalailusta, joissa kuluttaja on menettänyt autonomian ja markkinoija hallitsee täysin hänen päätöksiään. Sittemmin, asenteet ovat kehittyneet kohti realistisempaa optimismia, jossa neuromarkkinoinnin potentiaali tunnistetaan, mutta ei etsitä Eldoradoa. Myös dystooppisten tulevaisuuskuvien maalailusta on sittemmin luovuttu ja on siirrytty kohti kohtuullisempia eettisiä huolia, kuten aivotutkimusten turvallisuuden varmistaminen. Neuromarkkinoinnin alalle tyypillistä on korkeat sisääntulon kynnykset. Ajan kuluessa, kiinnostuksen kasvaessa neuromarkkinointia kohtaan, tutkijat ovat tehneet huomattavan määrän arvokasta työtä alentaakseen näitä sisääntulon esteitä. Tämän tutkielman tulosten valossa, kirjoittaja suosittelee johtajien käyttävän erityistä harkintaa halutessaan perustaa päätöksentekoaan neuromarkkinoinnin tarjoamiin tutkimustuloksiin tämän hetkisen empiirisen tutkimuksen hajanaisen ja vaihtelevan laadun vuoksi. Lisäksi, neuromarkkinointiin pohjautuvaa konsultointia tarjoavia yrityksiä on olemassa jo huomattava määrä. Kirjoittaja ei voi tämän tutkielman tulosten valossa suositella investointeja näihin palveluihin. Tieteenharjoittajille neuromarkkinointi ja kuluttajan aivotutkimus tarjoavat mielenkiintoisen tutkimusalan. Kirjoittaja suosittelee painopisteen siirtämistä kohti empiiristä tutkimusta, koska olemassa oleva teoreettinen kirjallisuus tarjoaa tarvittavan pohjan hyvälle empiiriselle tutkimukselle. Uudet empiirisen tutkimuksen tulokset lunastaisivat viimein neuromarkkinoinnin kauan tunnistetun potentiaalin ja siihen liittyvät odotukset