Data-driven multivariate and multiscale methods for brain computer interface

This thesis focuses on the development of data-driven multivariate and multiscale methods for brain computer interface (BCI) systems. The electroencephalogram (EEG), the most convenient means to measure neurophysiological activity due to its noninvasive nature, is mainly considered. The nonlinearity and nonstationarity inherent in EEG and its multichannel recording nature require a new set of data-driven multivariate techniques to estimate more accurately features for enhanced BCI operation. Also, a long term goal is to enable an alternative EEG recording strategy for achieving long-term and portable monitoring. Empirical mode decomposition (EMD) and local mean decomposition (LMD), fully data-driven adaptive tools, are considered to decompose the nonlinear and nonstationary EEG signal into a set of components which are highly localised in time and frequency. It is shown that the complex and multivariate extensions of EMD, which can exploit common oscillatory modes within multivariate (multichannel) data, can be used to accurately estimate and compare the amplitude and phase information among multiple sources, a key for the feature extraction of BCI system. A complex extension of local mean decomposition is also introduced and its operation is illustrated on two channel neuronal spike streams. Common spatial pattern (CSP), a standard feature extraction technique for BCI application, is also extended to complex domain using the augmented complex statistics. Depending on the circularity/noncircularity of a complex signal, one of the complex CSP algorithms can be chosen to produce the best classification performance between two different EEG classes. Using these complex and multivariate algorithms, two cognitive brain studies are investigated for more natural and intuitive design of advanced BCI systems. Firstly, a Yarbus-style auditory selective attention experiment is introduced to measure the user attention to a sound source among a mixture of sound stimuli, which is aimed at improving the usefulness of hearing instruments such as hearing aid. Secondly, emotion experiments elicited by taste and taste recall are examined to determine the pleasure and displeasure of a food for the implementation of affective computing. The separation between two emotional responses is examined using real and complex-valued common spatial pattern methods. Finally, we introduce a novel approach to brain monitoring based on EEG recordings from within the ear canal, embedded on a custom made hearing aid earplug. The new platform promises the possibility of both short- and long-term continuous use for standard brain monitoring and interfacing applications

Affective Brain-Computer Interfaces Neuroscientific Approaches to Affect Detection

The brain is involved in the registration, evaluation, and representation of emotional events, and in the subsequent planning and execution of adequate actions. Novel interface technologies – so-called affective brain-computer interfaces (aBCI) - can use this rich neural information, occurring in response to affective stimulation, for the detection of the affective state of the user. This chapter gives an overview of the promises and challenges that arise from the possibility of neurophysiology-based affect detection, with a special focus on electrophysiological signals. After outlining the potential of aBCI relative to other sensing modalities, the reader is introduced to the neurophysiological and neurotechnological background of this interface technology. Potential application scenarios are situated in a general framework of brain-computer interfaces. Finally, the main scientific and technological challenges that have to be solved on the way toward reliable affective brain-computer interfaces are discussed

Emotional Brain-Computer Interfaces

Research in Brain-computer interface (BCI) has significantly increased during the last few years. In addition to their initial role as assisting devices for the physically challenged, BCIs are now proposed for a wider range of applications. As in any HCI application, BCIs can also benefit from adapting their operation to the emotional state of the user. BCIs have the advantage of having access to brain activity which can provide signicant insight into the user's emotional state. This information can be utilized in two manners. 1) Knowledge of the inuence of the emotional state on brain activity patterns can allow the BCI to adapt its recognition algorithms, so that the intention of the user is still correctly interpreted in spite of signal deviations induced by the subject's emotional state. 2) The ability to recognize emotions can be used in BCIs to provide the user with more natural ways of controlling the BCI through affective modulation. Thus, controlling a BCI by recollecting a pleasant memory can be possible and can potentially lead to higher information transfer rates.\ud These two approaches of emotion utilization in BCI are elaborated in detail in this paper in the framework of noninvasive EEG based BCIs

On the analysis of EEG power, frequency and asymmetry in Parkinson's disease during emotion processing

Objective: While Parkinson’s disease (PD) has traditionally been described as a movement disorder, there is growing evidence of disruption in emotion information processing associated with the disease. The aim of this study was to investigate whether there are specific electroencephalographic (EEG) characteristics that discriminate PD patients and normal controls during emotion information processing. Method: EEG recordings from 14 scalp sites were collected from 20 PD patients and 30 age-matched normal controls. Multimodal (audio-visual) stimuli were presented to evoke specific targeted emotional states such as happiness, sadness, fear, anger, surprise and disgust. Absolute and relative power, frequency and asymmetry measures derived from spectrally analyzed EEGs were subjected to repeated ANOVA measures for group comparisons as well as to discriminate function analysis to examine their utility as classification indices. In addition, subjective ratings were obtained for the used emotional stimuli. Results: Behaviorally, PD patients showed no impairments in emotion recognition as measured by subjective ratings. Compared with normal controls, PD patients evidenced smaller overall relative delta, theta, alpha and beta power, and at bilateral anterior regions smaller absolute theta, alpha, and beta power and higher mean total spectrum frequency across different emotional states. Inter-hemispheric theta, alpha, and beta power asymmetry index differences were noted, with controls exhibiting greater right than left hemisphere activation. Whereas intra-hemispheric alpha power asymmetry reduction was exhibited in patients bilaterally at all regions. Discriminant analysis correctly classified 95.0% of the patients and controls during emotional stimuli. Conclusion: These distributed spectral powers in different frequency bands might provide meaningful information about emotional processing in PD patients

Emotional arousal modulates oscillatory correlates of targeted memory reactivation during NREM, but not REM sleep

Rapid eye movement (REM) sleep is considered to preferentially reprocess emotionally arousing memories. We tested this hypothesis by cueing emotional vs. neutral memories during REM and NREM sleep and wakefulness by presenting associated verbal memory cues after learning. Here we show that cueing during NREM sleep significantly improved memory for emotional pictures, while no cueing benefit was observed during REM sleep. On the oscillatory level, successful memory cueing during NREM sleep resulted in significant increases in theta and spindle oscillations with stronger responses for emotional than neutral memories. In contrast during REM sleep, solely cueing of neutral (but not emotional) memories was associated with increases in theta activity. Our results do not support a preferential role of REM sleep for emotional memories, but rather suggest that emotional arousal modulates memory replay and consolidation processes and their oscillatory correlates during NREM sleep

Does Breath Pattern Influence Reacting and Feeling?

This experimental within-participant reversal paradigm quantified effects of breath manipulation on emotional reactivity and inhibition. Participants were assessed for inhibitory ability and emotional reactivity at baseline and following three breathing conditions: controlled neutral, resonance frequency, and variable breathing; selected to assess a range of breathing behavior from anxious breathing, vegetative breathing, and meditative breathing. Emotional reactivity was elicited using the International Affective Picture System and inhibition utilizing a verbal Stop Signal task. Dependent variables for emotion induction included self-reported mood and arousal using the Self-Assessment Manikin of Valence and Arousal, and for inhibition was response time and accuracy. For twenty-six healthy participants, emotion induction demonstrated no statistical findings across breathing condition. However, for inhibition tasks, a significant reduction in inhibitory response time and increase in response accuracy was found following resonance frequency breathing. Breath manipulation effects inhibitory control and could be a tool for improving efficacy of behavioral therapies addressing aspects of inhibition

Optimal set of EEG features for emotional state classification and trajectory visualization in Parkinson's disease

In addition to classic motor signs and symptoms, individuals with Parkinson's disease (PD) are characterized by emotional deficits. Ongoing brain activity can be recorded by electroencephalograph (EEG) to discover the links between emotional states and brain activity. This study utilized machine-learning algorithms to categorize emotional states in PD patients compared with healthy controls (HC) using EEG. Twenty non-demented PD patients and 20 healthy age-, gender-, and education level-matched controls viewed happiness, sadness, fear, anger, surprise, and disgust emotional stimuli while fourteen-channel EEG was being recorded. Multimodal stimulus (combination of audio and visual) was used to evoke the emotions. To classify the EEG-based emotional states and visualize the changes of emotional states over time, this paper compares four kinds of EEG features for emotional state classification and proposes an approach to track the trajectory of emotion changes with manifold learning. From the experimental results using our EEG data set, we found that (a) bispectrum feature is superior to other three kinds of features, namely power spectrum, wavelet packet and nonlinear dynamical analysis; (b) higher frequency bands (alpha, beta and gamma) play a more important role in emotion activities than lower frequency bands (delta and theta) in both groups and; (c) the trajectory of emotion changes can be visualized by reducing subject-independent features with manifold learning. This provides a promising way of implementing visualization of patient's emotional state in real time and leads to a practical system for noninvasive assessment of the emotional impairments associated with neurological disorders

Timing the senses and sensing the time: Individual differences in subjective duration

Although the experimental investigation of the perception of time dates back to around 1864 (Lejeune & Wearden, 2009) the reasons for variation between individuals are still poorly understood. Advancing in this area has been identified as one of the most important issues for the modern investigation of time perception (Hancock & Block, 2012). Although various individual differences in time perception have been identified, these are usually based on comparisons between different groups depending on characteristics such as age or gender; clinical conditions such as schizophrenia or autism; or induced differences such as temperature or pharmacology. In this thesis we seek to investigate whether, and how, intrinsic individual differences in sensory processing within the general population influence individual timing behaviour. This is accomplished over four experiments. The first seeks a relationship between the EEG alpha rhythm, which is strongly associated with visual and audio-visual temporal integration, and behaviour on four timing tasks; while trends are in the expected directions no significant relationship emerges. The second responds to this null result by seeking a relationship between audio-visual integration and estimated, sub-second, durations in a purely behavioural paradigm, in this case the results show a significant association. The final two experiments are concerned with individual differences in interoceptive accuracy (sensitivity to ones own bodily signals), and how these influence the effect of arousal on time judgement. The first experiment, using supra-second durations, finds no effects, but the second, using shorter durations (under 1200ms) and addressing some methodological concerns, does find a significant moderation, by interoception, of the effect of emotional arousal upon time. We conclude that this provides substantial evidence that exteroceptive and interoceptive primary sensory processes play a role in individual variation in timing tasks, a finding that provides many further avenues of investigation