Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges

In recent years, new research has brought the field of EEG-based Brain-Computer Interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely,“Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user-machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human-computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices

Improved physiological noise regression in fNIRS: a multimodal extension of the General Linear Model using temporally embedded Canonical Correlation Analysis

For the robust estimation of evoked brain activity from functional Near-Infrared Spectroscopy (fNIRS) signals, it is crucial to reduce nuisance signals from systemic physiology and motion. The current best practice incorporates short-separation (SS) fNIRS measurements as regressors in a General Linear Model (GLM). However, several challenging signal characteristics such as non-instantaneous and non-constant coupling are not yet addressed by this approach and additional auxiliary signals are not optimally exploited. We have recently introduced a new methodological framework for the unsupervised multivariate analysis of fNIRS signals using Blind Source Separation (BSS) methods. Building onto the framework, in this manuscript we show how to incorporate the advantages of regularized temporally embedded Canonical Correlation Analysis (tCCA) into the supervised GLM. This approach allows flexible integration of any number of auxiliary modalities and signals. We provide guidance for the selection of optimal parameters and auxiliary signals for the proposed GLM extension. Its performance in the recovery of evoked HRFs is then evaluated using both simulated ground truth data and real experimental data and compared with the GLM with short-separation regression. Our results show that the GLM with tCCA significantly improves upon the current best practice, yielding significantly better results across all applied metrics: Correlation (HbO max. +45%), Root Mean Squared Error (HbO max. -55%), F-Score (HbO up to 3.25-fold) and p-value as well as power spectral density of the noise floor. The proposed method can be incorporated into the GLM in an easily applicable way that flexibly combines any available auxiliary signals into optimal nuisance regressors. This work has potential significance both for conventional neuroscientific fNIRS experiments as well as for emerging applications of fNIRS in everyday environments, medicine and BCI, where high Contrast to Noise Ratio is of importance for single trial analysis.Published versio

Predictive decoding of neural data

In the last five decades the number of techniques available for non-invasive functional imaging has increased dramatically. Researchers today can choose from a variety of imaging modalities that include EEG, MEG, PET, SPECT, MRI, and fMRI. This doctoral dissertation offers a methodology for the reliable analysis of neural data at different levels of investigation. By using statistical learning algorithms the proposed approach allows single-trial analysis of various neural data by decoding them into variables of interest. Unbiased testing of the decoder on new samples of the data provides a generalization assessment of decoding performance reliability. Through consecutive analysis of the constructed decoder\u27s sensitivity it is possible to identify neural signal components relevant to the task of interest. The proposed methodology accounts for covariance and causality structures present in the signal. This feature makes it more powerful than conventional univariate methods which currently dominate the neuroscience field. Chapter 2 describes the generic approach toward the analysis of neural data using statistical learning algorithms. Chapter 3 presents an analysis of results from four neural data modalities: extracellular recordings, EEG, MEG, and fMRI. These examples demonstrate the ability of the approach to reveal neural data components which cannot be uncovered with conventional methods. A further extension of the methodology, Chapter 4 is used to analyze data from multiple neural data modalities: EEG and fMRI. The reliable mapping of data from one modality into the other provides a better understanding of the underlying neural processes. By allowing the spatial-temporal exploration of neural signals under loose modeling assumptions, it removes potential bias in the analysis of neural data due to otherwise possible forward model misspecification. The proposed methodology has been formalized into a free and open source Python framework for statistical learning based data analysis. This framework, PyMVPA, is described in Chapter 5

a methodological approach

In natural environments, visual and auditory stimulation elicit responses across a large set of brain regions in a fraction of a second, yielding representations of the multimodal scene and its properties. The rapid and complex neural dynamics underlying visual and auditory information processing pose major challenges to human cognitive neuroscience. Brain signals measured non-invasively are inherently noisy, the format of neural representations is unknown, and transformations between representations are complex and often nonlinear. Further, no single non-invasive brain measurement technique provides a spatio-temporally integrated view. In this opinion piece, we argue that progress can be made by a concerted effort based on three pillars of recent methodological development: (i) sensitive analysis techniques such as decoding and cross-classification, (ii) complex computational modelling using models such as deep neural networks, and (iii) integration across imaging methods (magnetoencephalography/electroencephalography, functional magnetic resonance imaging) and models, e.g. using representational similarity analysis. We showcase two recent efforts that have been undertaken in this spirit and provide novel results about visual and auditory scene analysis. Finally, we discuss the limits of this perspective and sketch a concrete roadmap for future research

EpiGauss : caracterização espacio-temporal da actividade cerebral em epilepsia

Doutoramento em Engenharia ElectrotécnicaA epilepsia é uma patologia cerebral que afecta cerca de 0,5% da população mundial. Nas epilepsias focais, o principal objectivo clínico é a localização da zona epileptogénica (área responsável pelas crises), uma informação crucial para uma terapêutica adequada. Esta tese é centrada na caracterização da actividade cerebral electromagnética do cérebro epiléptico. As contribuições nesta área, entre a engenharia e neurologia clínica, são em duas direcções. Primeiro, mostramos que os conceitos associados às pontas podem ser imprecisos e não ter uma definição objectiva, tornando necessária uma reformulação de forma a definir uma referência fiável em estudos relacionados com a análise de pontas. Mostramos que as características das pontas em EEG são estatisticamente diferentes das pontas em MEG. Esta constatação leva a concluir que a falta de objectividade na definição de ponta na literatura pode induzir utilizações erradas de conceitos associados ao EEG na análise de MEG. Também verificamos que o uso de conjuntos de detecções de pontas efectuadas por especialistas (MESS) como referência pode fornecer resultados enganadores quando apenas baseado em critérios de consenso clínico, nomeadamente na avaliação da sensibilidade e especificidade de métodos computorizados de detecção de pontas Em segundo lugar, propomos o uso de métodos estatísticos para ultrapassar a falta de precisão e objectividade das definições relacionadas com pontas. Propomos um novo método de neuroimagem suportado na caracterização de geradores electromagnéticos – EpiGauss – baseado na análise individual dos geradores de eventos do EEG que explora as suas estruturas espacio-temporais através da análise de “clusters”. A aplicação de análise de “clusters” à análise geradores de eventos do EEG tem como objectivo usar um método não supervisionado, para encontrar estruturas espacio-temporais dps geradores relevantes. Este método, como processo não supervisionado, é orientado a utilizadores clínicos e apresenta os resultados sob forma de imagens médicas com interpretação similar a outras técnicas de imagiologia cerebral. Com o EpiGauss, o utilizador pode determinar a localização estatisticamente mais provável de geradores, a sua estabilidade espacial e possíveis propagações entre diferente áreas do cérebro. O método foi testado em dois estudos clínicos envolvendo doentes com epilepsia associada aos hamartomas hipotalâmicos e o outro com doentes com diagnóstico de epilepsia occipital. Em ambos os estudos, o EpiGauss foi capaz de identificar a zona epileptogénica clínica, de forma consistente com a história e avaliação clínica dos neurofisiologistas, fornecendo mais informação relativa à estabilidade dos geradores e possíveis percursos de propagação da actividade epileptogénica contribuindo para uma melhor caracterização clínica dos doentes. A conclusão principal desta tese é que o uso de técnicas não supervisionadas, como a análise de “clusters”, associadas as técnicas não-invasivas de EMSI, pode contribuir com um valor acrescido no processo de diagnóstico clínico ao fornecer uma caracterização objectiva e representação visual de padrões complexos espaciotemporais da actividade eléctrica epileptogénica.Epilepsy is a brain pathology that affects 0.5% of the world population. In focal epilepsies, the main clinical objective is the localization of the epileptogenic zone (brain area responsible for the epileptic seizures – EZ), a key information to decide an adequate therapeutic approach. This thesis is centred on electromagnetic activity characterization of the epileptic brain. Our contribution to this boundary area between engineering and clinical neurology is two-folded. First we show that spike related clinical concepts can be unprecise and some do not have objective definitions making necessary a reformulation in order to have a reliable reference in spike related studies. We show that EEG spike wave quantitative features are statistically different from their MEG counterparts. This finding leads to the conclusion that the lack of objective spike feature definitions in the literature can induce the wrong usage of EEG feature definition in MEG analysis. We also show that the use of multi-expert spike selections sets (MESS) as gold standard, although clinically useful, may be misleading whenever defined solely in terms of clinical agreement criteria, namely as references for automatic spike detection algorithms in sensitivity and specificity method analysis. Second, we propose the use of statistical methods to overcome some lack of precision and objectivity in spike related definitions. In this context, we propose a new ElectroMagnetic Source Imaging (EMSI) method – EpiGauss – based on cluster analysis that explores both spatial and temporal information contained in individual events sources analysis characterisation. This automatic cluster method for the analysis of spike related electric generators based in EEG is used to provide an unsupervised tool to find their relevant spatio-temporal structures. This method enables a simple unsupervised procedure aimed for clinical users and presents its results in an intuitive representation similar to other brain imaging techniques. With EpiGauss, the user is able to determine statistically probable source locations, their spatial stability and propagation patterns between different brain areas. The method was tested in two different clinical neurophysiology studies, one with a group of Hypothalamic Hamartomas and another with a group of Occipital Epilepsy patients. In both studies EpiGauss identified the clinical epileptogenic zone, consistent with the clinical background and evaluation of neurophysiologists, providing further information on stability of source locations and their probable propagation pathways that enlarges their clinical interpretation. This thesis main conclusion is that the use of unsupervised techniques, such as clustering, associated with EMSI non-invasive techniques, can bring an added value in clinical diagnosis process by providing objective and visual representation of complex epileptic brain spatio-temporal activity patterns

Revealing Distinct Neural Signatures in Magnetoencephalography with Hidden Markov Models

Magnetoencephalography (MEG) is a functional neuroimaging method which measures the magnetic fields produced by neural communication in the brain. Specifically, the fields induced by dendritic current flow in assemblies of pyramidal neurons. Because these magnetic fields are generated directly by brain electrophysiology, and are mostly unperturbed by the skull, MEG data are rich in spatial and temporal information. This thesis is chiefly concerned with interpreting these data in a way that produces useful results whilst minimising bias. Hidden Markov modelling (HMM) is a robust statistical method which has been applied to fields as diverse as speech recognition and financial market prediction. It parses data into a number of ‘hidden states’, each with their own unique characteristics, in an unsupervised way. Because it is data-driven, it can create a model unique to each participant’s brain activity and specific to each task. In addition, the HMM framework itself is flexible so it can be applied to both sensor and source-space data and can be applied to multiple channels (multivariate) or to a single time course (univariate). Choice of an observation model allows states to be characterised by amplitude, spatial, or spectral content depending on the research question. The aim of this thesis is to apply hidden Markov modelling (HMM) to whole head MEG data to identify repeated patterns of transient neural activity occurring throughout the brain. Once these patterns were identified, the interaction between these short ‘bursts’ of activity across the cortex was established which provided a unique measure of functional connectivity. Three studies were undertaken: The role of transient spectral bursts in MEG functional connectivity: In recent years, the smoothly varying neural oscillations often studied in MEG (such as those trial-averaged responses in the traditional neurophysiological (such as alpha/beta) frequency bands) have been shown to be made up of single-trial high-amplitude ‘bursts’ of activity. These bursts can be observed in the beta frequency band and are therefore often referred to as beta bursts. In this study, a novel time-delay embedded HMM was used to identif bursts in broadband data based on their spectral content for MEG data from 66 healthy adult participants. The burst amplitude, duration and frequency of occurrence were characterised across the cortex in resting state data, and in a motor task the classic movement-related beta desynchronisation and post movement beta rebound were shown to be made up of changes in burst occurrence. A novel functional connectivity metric was then introduced based on the coincidence of bursts from distal brain regions, allowing the known beta band functional connectome to be reproduced. Bursts coincident across spatially separate brain regions were also shown to correspond to periods of heightened coherence, lending evidence to the communication by coherence (Fries 2005, 2015) hypothesis. Post-stimulus responses across the cortex: During a motor task, both primary (during stimulation) and post stimulus responses (PSR) can be observed. These are well characterised in the literature, but little is known about their functional significance. The PSR in particular is modified in a range of seemingly unrelated neurological conditions with variable symptoms, such as schizophrenia (Robson et al. 2016), autism spectrum disorder (Gaetz et al. 2020) and multiple sclerosis (Barratt et al. 2017), indicating that the PSR is a fundamental neurophysiological process, the disturbance of which has implications on both healthy and pathological brain function. This work therefore tested the hypothesis that the PSR is present across the cortex. MEG data were acquired and analysed from two experiments with 15 healthy adult volunteers each – the first was a right-hand grip task with visual feedback, the second involved passive left visual field stimulation. Both experiments varied stimulus duration (2s, 5s and 10s) with a 30s rest-period between trials to allow characterisation of the full PSR. A univariate 3-state time-delay-embedded hidden Markov model (HMM) was used to characterise the spatial distributions of the primary and PSR across the cortex for both tasks. Results showed that for both tasks, the primary response state was more bilateral over the sensorimotor or visual areas (depending on task) where the PSR state was more unilateral and confined to the contralateral sensorimotor or visual areas (again, dependant on task). A state coincidence metric was then used to investigate the integration of the primary and PSR states across brain regions as a measure of task-related functional connectivity. Hidden Markov modelling of the interictal brain: Epilepsy is a highly heterogeneous disease with variations in the temporal morphology and localisation of epileptiform activity across patients. Unsupervised machine learning techniques like the HMM allow us to take into account this variability and ensure that every model is tailored to each individual. In this work, a multivariate time-delay embedded HMM was used to identify brain states based on their spatial and spectral properties in sensor-level MEG data acquired as part of standard clinical care for patients at the Children’s Hospital of Philadelphia. State allocations were used together with a linearly constrained minimum variance (LCMV) beamformer to produce a 3D map of state variance, hence localising probable epileptogenic foci. Clinical MEG epilepsy data are routinely analysed by excess kurtosis mapping (EKM) and so the performance of the HMM was assessed against this for three patient groups, each with increasingly complex epilepsy manifestation (10 patients in total). The difference in localization of epileptogenic foci for the two methods was 7 ± 2mm (mean ± SD over all 10 patients); and 94 ± 13% of EKM temporal markers were matched by an HMM state visit. It is therefore clear that this method localizes epileptogenic areas in agreement with EKM and in patients with more than one focus the HMM provides additional information about the relationship between them

Aligning computer and human visual representations

Both computer vision and human visual system target the same goal: to accomplish visual tasks easily via a set of representations. In this thesis, we study to what extent representations from computer vision models align to human visual representations. To study this research question we used an interdisciplinary approach, integrating methods from psychology, neuroscience and computer vision. Such an approach is aimed to provide new insight in the understanding of human visual representations. In the four chapters of the thesis, we tested computer vision models against brain data obtained with electro-encephalography (EEG) and functional magnetic resonance imaging (fMRI). The main findings can be summarized as follows; 1) computer vision models with one or two computational stages correlate to visual representations of intermediate complexity in the human brain, 2) models with multiple computational stages correlate best to the hierarchy of representations in the human visual system, 3) computer vision models do not align one-to-one to the temporal hierarchy of representations in the visual cortex and 4) not only visual but also semantic representations correlate to representations in the human visual system

Development and Evaluation of Data Processing Techniques in Magnetoencephalography

With MEG, the tiny magnetic fields produced by neuronal currents within the brain can be measured completely non-invasively. But the signals are very small (~100 fT) and often obscured by spontaneous brain activity and external noise. So, a recurrent issue in MEG data analysis is the identification and elimination of this unwanted interference within the recordings. Various strategies exist to meet this purpose. In this thesis, two of these strategies are scrutinized in detail. The first is the commonly used procedure of averaging over trials which is a successfully applied data reduction method in many neurocognitive studies. However, the brain does not always respond identically to repeated stimuli, so averaging can eliminate valuable information. Alternative approaches aiming at single trial analysis are difficult to realize and many of them focus on temporal patterns. Here, a compromise involving random subaveraging of trials and repeated source localization is presented. A simulation study with numerous examples demonstrates the applicability of the new method. As a result, inferences about the generators of single trials can be drawn which allows deeper insight into neuronal processes of the human brain. The second technique examined in this thesis is a preprocessing tool termed Signal Space Separation (SSS). It is widely used for preprocessing of MEG data, including noise reduction by suppression of external interference, as well as movement correction. Here, the mathematical principles of the SSS series expansion and the rules for its application are investigated. The most important mathematical precondition is a source-free sensor space. Using three data sets, the influence of a violation of this convergence criterion on source localization accuracy is demonstrated. The analysis reveals that the SSS method works reliably, even when the convergence criterion is not fully obeyed. This leads to utilizing the SSS method for the transformation of MEG data to virtual sensors on the scalp surface. Having MEG data directly on the individual scalp surface would alleviate sensor space analysis across subjects and comparability with EEG. A comparison study of the transformation results obtained with SSS and those produced by inverse and subsequent forward computation is performed. It shows strong dependence on the relative position of sources and sensors. In addition, the latter approach yields superior results for the intended purpose of data transformation

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task