Deep fusion of multi-channel neurophysiological signal for emotion recognition and monitoring

How to fuse multi-channel neurophysiological signals for emotion recognition is emerging as a hot research topic in community of Computational Psychophysiology. Nevertheless, prior feature engineering based approaches require extracting various domain knowledge related features at a high time cost. Moreover, traditional fusion method cannot fully utilise correlation information between different channels and frequency components. In this paper, we design a hybrid deep learning model, in which the 'Convolutional Neural Network (CNN)' is utilised for extracting task-related features, as well as mining inter-channel and inter-frequency correlation, besides, the 'Recurrent Neural Network (RNN)' is concatenated for integrating contextual information from the frame cube sequence. Experiments are carried out in a trial-level emotion recognition task, on the DEAP benchmarking dataset. Experimental results demonstrate that the proposed framework outperforms the classical methods, with regard to both of the emotional dimensions of Valence and Arousal

Markers of criticality in phase synchronization

The concept of the brain as a critical dynamical system is very attractive because systems close to criticality are thought to maximize their dynamic range of information processing and communication. To date, there have been two key experimental observations in support of this hypothesis: (i) neuronal avalanches with power law distribution of size and (ii) long-range temporal correlations (LRTCs) in the amplitude of neural oscillations. The case for how these maximize dynamic range of information processing and communication is still being made and because a significant substrate for information coding and transmission is neural synchrony it is of interest to link synchronization measures with those of criticality. We propose a framework for characterizing criticality in synchronization based on an analysis of the moment-to-moment fluctuations of phase synchrony in terms of the presence of LRTCs. This framework relies on an estimation of the rate of change of phase difference and a set of methods we have developed to detect LRTCs. We test this framework against two classical models of criticality (Ising and Kuramoto) and recently described variants of these models aimed to more closely represent human brain dynamics. From these simulations we determine the parameters at which these systems show evidence of LRTCs in phase synchronization. We demonstrate proof of principle by analysing pairs of human simultaneous EEG and EMG time series, suggesting that LRTCs of corticomuscular phase synchronization can be detected in the resting state and experimentally manipulated. The existence of LRTCs in fluctuations of phase synchronization suggests that these fluctuations are governed by non-local behavior, with all scales contributing to system behavior. This has important implications regarding the conditions under which one should expect to see LRTCs in phase synchronization. Specifically, brain resting states may exhibit LRTCs reflecting a state of readiness facilitating rapid task-dependent shifts toward and away from synchronous states that abolish LRTCs

Appreciating the Performance of Neuroscience Mining in NeuroIS research: A Case Study on Consumer's Product Perceptions in the Two UI Modes—Dark UI vs. Light UI

The goal of the current study was to provide information on the potential of neuroscience mining (NSM) for comprehending NeuroIS paradigms. NSM is an interdisciplinary field that combines neuroscience and business mining, which is the application of big data analytics, computational social science, and other fields to business problems. Therefore, NSM makes it possible to apply predictive models to NeuroIS datasets, such as machine learning and deep learning, to find intricate patterns that are hidden by conventional regression-based analysis. We predicted 28 individual EEG power spectra separated brainwave data using a Random Forest (RF) model. Next, we used NSM to precisely predict how consumers would perceive a product online, depending on whether a light or dark user interface (UI) mode was being used. The model was then used to extract more precise results that could not be obtained using more conventional linear-based analytical models using sensitivity analysis. The benefits of using NSM in NeuroIS research are as follows: (1) it can relieve the burden of the three-horned dilemma described by Runkel and McGrath; (2) it can enable more temporal data to be directly analyzed on the target variables; and (3) sensitivity analysis can be performed on a condition/individual basis, strengthening the rigor of findings by reducing sample bias that can be lost in grand averaging of data when analyzed with methods like GLM

Improving classification of epileptic and non-epileptic EEG events by feature selection

This is the Accepted Manuscript version of the following article: E. Pippa, et al, “Improving classification of epileptic and non-epileptic EEG events by feature selection”, Neurocomputing, Vol. 171: 576-585, July 2015. The final published version is available at: http://www.sciencedirect.com/science/article/pii/S0925231215009509?via%3Dihub Copyright © 2015 Elsevier B.V.Correctly diagnosing generalized epileptic from non-epileptic episodes, such as psychogenic non epileptic seizures (PNES) and vasovagal or vasodepressor syncope (VVS), despite its importance for the administration of appropriate treatment, life improvement of the patient, and cost reduction for patient and healthcare system, is rarely tackled in the literature. Usually clinicians differentiate between generalized epileptic seizures and PNES based on clinical features and video-EEG. In this work, we investigate the use of machine learning techniques for automatic classification of generalized epileptic and non-epileptic events based only on multi-channel EEG data. For this purpose, we extract the signal patterns in the time domain and in the frequency domain and then combine all features across channels to characterize the spatio-temporal manifestation of seizures. Several classification algorithms are explored and evaluated on EEG epochs from 11 subjects in an inter-subject cross-validation setting. Due to large number of features feature ranking and selection is performed prior to classification using the ReliefF ranking algorithm within two different voting strategies. The classification models using feature subsets, achieved higher accuracy compared to the models using all features reaching 95% (Bayesian Network), 89% (Random Committee) and 87% (Random Forest) for binary classification (epileptic versus non-epileptic). The results demonstrate the competitiveness of this approach as opposed to previous methods.Peer reviewe

Graph analysis of functional brain networks: practical issues in translational neuroscience

The brain can be regarded as a network: a connected system where nodes, or units, represent different specialized regions and links, or connections, represent communication pathways. From a functional perspective communication is coded by temporal dependence between the activities of different brain areas. In the last decade, the abstract representation of the brain as a graph has allowed to visualize functional brain networks and describe their non-trivial topological properties in a compact and objective way. Nowadays, the use of graph analysis in translational neuroscience has become essential to quantify brain dysfunctions in terms of aberrant reconfiguration of functional brain networks. Despite its evident impact, graph analysis of functional brain networks is not a simple toolbox that can be blindly applied to brain signals. On the one hand, it requires a know-how of all the methodological steps of the processing pipeline that manipulates the input brain signals and extract the functional network properties. On the other hand, a knowledge of the neural phenomenon under study is required to perform physiological-relevant analysis. The aim of this review is to provide practical indications to make sense of brain network analysis and contrast counterproductive attitudes

The Impact of Math Anxiety on Working Memory:A Cortical Activations and Cortical Functional Connectivity EEG Study

Mathematical anxiety (MA) is defined as a feeling of tension, apprehension, or fear that interferes with mathematical performance in various daily or academic situations. Cognitive consequences of MA have been studied a lot and revealed that MA seriously affects solving the complex problem due to the corruption of working memory (WM). The corruption of WM caused by MA is well documented in behavioral level, but the involved neurophysiological processes have not been properly addressed, despite the recent attention drawn on the neural basis of MA. This is the second part of our study that intents to investigate the neurophysiological aspects of MA and its implications to WM. In the first study, we saw how MA affects the early stages of numeric stimuli processes as the WM indirectly using event-related potentials in scalp electroencephalographic (EEG) signals. This paper goes one step further to investigate the cortical activations, obtained by the multichannel EEG recordings as well as the cortical functional networks in three WM tasks with increasing difficulty. Our results indicate that the high-math anxious (HMA) group activated more areas linked with negative emotions, pain, and fear, while the low-math anxious (LMA) group activated regions related to the encoding and retrieval processes of the WM. Functional connectivity analysis also reveals that the LMAs' brain has got more structured cortical networks with increased connectivity in areas related to WM, such as the frontal cortex, while the HMAs' brain has a more diffused and unstructured network, superimposing the evidence that the structured processes of WM are corrupted

Development of EEG-based technologies for the characterization and treatment of neurological diseases affecting the motor function

This thesis presents a set of studies applying signal processing and data mining techniques in real-time working systems to register, characterize and condition the movement-related cortical activity of healthy subjects and of patients with neurological disorders affecting the motor function. Patients with two of the most widespread neurological affections impairing the motor function are considered here: patients with essential tremor and patients who have suffered a cerebro-vascular accident. The different chapters in the presented thesis show results regarding the normal cortical activity associated with the planning and execution of motor actions with the upper-limb, and the pathological activity related to the patients' motor dysfunction (measurable with muscle electrodes or movement sensors). The initial chapters of the book present i) a revision of the basic concepts regarding the role of the cerebral cortex in the motor control and the way in which the electroencephalographic activity allows its analysis and conditioning, ii) a study on the cortico-muscular interaction at the tremor frequency in patients with essential tremor under the effects of a drug reducing their tremor, and finally iii) a study based on evolutionary algorithms that aims to identify cortical patterns related to the planning of a number of motor tasks performed with a single arm. In the second half of the thesis book, two brain-computer interface systems to be used in rehabilitation scenarios with essential tremor patients and with patients with a stroke are proposed. In the first system, the electroencephalographic activity is used to anticipate voluntary movement actions, and this information is integrated in a multimodal platform estimating and suppressing the pathological tremors. In the second case, a conditioning paradigm for stroke patients based on the identification of the motor intention with temporal precision is presented and tested with a cohort of four patients along a month during which the patients undergo eight intervention sessions. The presented thesis has yielded advances from both the technological and the scientific points of view in all studies proposed. The main contributions from the technological point of view are: ¿ The design of an integrated upper-limb platform working in real-time. The platform was designed to acquire information from different types of noninvasive sensors (EEG, EMG and gyroscopic sensors) characterizing the planning and execution of voluntary movements. The platform was also capable of processing online the acquired data and generating an electrical feedback. ¿ The development of signal processing and classifying techniques adapted to the kind of signal recorded in the two kinds of patients considered in this thesis (patients with essential tremor and patients with a stroke) and to the requirements of online processing and real-time single-trial function desired for BCI applications. Especially in this regard, an original methodology to detect onsets of voluntary movements using slow cortical potentials and cortical rhythms has been presented. ¿ The design and validation in real-time of asynchronous BCI systems using motor planning EEG segments to anticipate or detect when patients begin a voluntary movement with the upper-limb. ¿ The proof of concept of the advantages of an EEG system integrated in a multimodal human-robot interface architecture that constitutes the first multimodal interface using the combined acquisition of EEG, EMG and gyroscopic data, which allows the concurrent characterization of different parts of the body associated with the execution of a movement. The main scientific contributions of this thesis are: ¿ The study of the EEG-based anticipation of voluntary movements presented in Chapter 5 of the thesis was the first demonstration (to the author's knowledge) of the capacity of the EEG signal to provide reliable movement predictions based on single-trial classification of online data of healthy subjects and ET patients. This study also provides, for the first time, the results of a BCI system tested in ET patients and it represents an original approach to BCI applications for this group of patients. ¿ It has been presented the first neurophysiological study using EEG and EMG data to analyze the effects of a drug on cortical activity and tremors of patients with ET. In addition, the obtained results have shown for the first time that a significant correlation exists between the dynamics of specific cortical oscillations and pathological tremor manifestation as a consequence of the drug effects. ¿ It has been proposed for the first time an experiment to inspect whether the EEG signal carries enough information to classify up to seven different tasks performed with a single limb. Both the methodology applied and the validation procedure are also innovative in this sort of studies. ¿ It has been demonstrated for the first time the relevance of combining different cortical sources of information (such as BP and ERD) to estimate the initiation of voluntary movements with the upper-limb. In this line, special relevance may be given to the positive results achieved with stroke patients, improving the results presented by similar previous EEG-based studies by other research groups. It has also been proposed for the first time an upper-limb intervention protocol for stroke patients using BP and ERD patterns to provide proprioceptive feedback tightly associated with the patients' expectations of movement. The effects of the proposed intervention have been studied with a small group of patients