Spatial Filtering Pipeline Evaluation of Cortically Coupled Computer Vision System for Rapid Serial Visual Presentation

Rapid Serial Visual Presentation (RSVP) is a paradigm that supports the application of cortically coupled computer vision to rapid image search. In RSVP, images are presented to participants in a rapid serial sequence which can evoke Event-related Potentials (ERPs) detectable in their Electroencephalogram (EEG). The contemporary approach to this problem involves supervised spatial filtering techniques which are applied for the purposes of enhancing the discriminative information in the EEG data. In this paper we make two primary contributions to that field: 1) We propose a novel spatial filtering method which we call the Multiple Time Window LDA Beamformer (MTWLB) method; 2) we provide a comprehensive comparison of nine spatial filtering pipelines using three spatial filtering schemes namely, MTWLB, xDAWN, Common Spatial Pattern (CSP) and three linear classification methods Linear Discriminant Analysis (LDA), Bayesian Linear Regression (BLR) and Logistic Regression (LR). Three pipelines without spatial filtering are used as baseline comparison. The Area Under Curve (AUC) is used as an evaluation metric in this paper. The results reveal that MTWLB and xDAWN spatial filtering techniques enhance the classification performance of the pipeline but CSP does not. The results also support the conclusion that LR can be effective for RSVP based BCI if discriminative features are available

Spatial Detection of Multiple Movement Intentions from SAM-Filtered Single-Trial MEG for a high performance BCI

The objective of this study is to test whether human intentions to sustain or cease movements in right and left hands can be decoded reliably from spatially filtered single trial magneto-encephalographic (MEG) signals. This study was performed using motor execution and motor imagery movements to achieve a potential high performance Brain-Computer interface (BCI). Seven healthy volunteers, naïve to BCI technology, participated in this study. Signals were recorded from 275-channel MEG and synthetic aperture magnetometry (SAM) was employed as the spatial filter. The four-class classification for natural movement intentions was performed offline; Genetic Algorithm based Mahalanobis Linear Distance (GA-MLD) and direct-decision tree classifier (DTC) techniques were adopted for the classification through 10-fold cross-validation. Through SAM imaging, strong and distinct event related desynchronisation (ERD) associated with sustaining, and event related synchronisation (ERS) patterns associated with ceasing of hand movements were observed in the beta band (15 - 30 Hz). The right and left hand ERD/ERS patterns were observed on the contralateral hemispheres for motor execution and motor imagery sessions. Virtual channels were selected from these cortical areas of high activity to correspond with the motor tasks as per the paradigm of the study. Through a statistical comparison between SAM-filtered virtual channels from single trial MEG signals and basic MEG sensors, it was found that SAM-filtered virtual channels significantly increased the classification accuracy for motor execution (GA-MLD: 96.51 ± 2.43 %) as well as motor imagery sessions (GA-MLD: 89.69 ± 3.34%). Thus, multiple movement intentions can be reliably detected from SAM-based spatially-filtered single trial MEG signals. MEG signals associated with natural motor behavior may be utilized for a reliable high-performance brain-computer interface (BCI) and may reduce long-term training compared with conventional BCI methods using rhythm control. This may prove tremendously helpful for patients suffering from various movement disorders to improve their quality of life

Decoding steady-state visual evoked potentials from electrocorticography

We report on a unique electrocorticography (ECoG) experiment in which Steady-State Visual Evoked Potentials (SSVEPs) to frequency-and phase-tagged stimuli were recorded from a large subdural grid covering the entire right occipital cortex of a human subject. The paradigm is popular in EEG-based Brain Computer Interfacing where selectable targets are encoded by different frequency-and/or phase-tagged stimuli. We compare the performance of two state-of-the-art SSVEP decoders on both ECoG-and scalp-recorded EEG signals, and show that ECoG-based decoding is more accurate for very short stimulation lengths (i.e., less than 1 s). Furthermore, whereas the accuracy of scalp-EEG decoding bene fi ts from a multi-electrode approach, to address interfering EEG responses and noise, ECoG decoding enjoys only a marginal improvement as even a single electrode, placed over the posterior part of the primary visual cortex, seems to suf fi ce. This study shows, for the fi rst time, that EEG-based SSVEP decoders can in principle be applied to ECoG, and can be expected to yield faster decoding speeds using less electrodes

Reconstruction of cortical sources activities for online classification of electroencephalographic signals.

International audienceWe compare the results given by different methods to reconstruct cortical sources activity in order to classify EEG in real time. Two motor imagery experiments were performed. The aim was to retrieve from 1-second windows of signal which motor imagery task the subjects were performing. The use of cortical activity reconstruction was compared to Laplacian filtering, which is often used in BCI. A recursive algorithm using Student's t-test was used to select relevant cortical sources. The Beamformer method led to an improvement of the classification for the first experiment, which included six motor imagery tasks. The weighted Minimum-Norm method required the use of a specific head model, extracted from the subject's MRI, to improve the classification. It then gave the best results on the second experiment, achieving a classification rate of 77% compared to 71% for direct use of electrode data and 75% for Laplacian filtering and Beamformer

MEG:hen perustuvan aivo-tietokone -käyttöliittymän kehitys

Brain–computer interfaces (BCI) have recently gained interest both in basic neuroscience and clinical interventions. The majority of noninvasive BCIs measure brain activity with electroencephalography (EEG). However, the real-time signal analysis and decoding of brain activity suffer from low signal-to-noise ratio and poor spatial resolution of EEG. These limitations could be overcome by using magnetoencephalography (MEG) as an alternative measurement modality. The aim of this thesis is to develop an MEG-based BCI for decoding hand motor imagery, which could eventually serve as a therapeutic method for patients recovering from e.g. cerebral stroke. Here, machine learning methods for decoding motor imagery -related brain activity are validated with healthy subjects’ MEG measurements. The first part of the thesis (Study I) involves a comparison of feature extraction methods for classifying left- vs right-hand motor imagery (MI), and MI vs rest. It was found that spatial filtering and further extraction of bandpower features yield better classification accuracy than time–frequency features extracted from parietal gradiometers. Furthermore, prior spatial filtering improved the discrimination capability of time–frequency features. The training data for a BCI is typically collected in the beginning of each measurement session. However, as this can be time-consuming and exhausting for the subject, the training data from other subjects’ measurements could be used as well. In the second part of the thesis (Study II), methods for across-subject classification of MI were compared. The results showed that a classifier based on multi-task learning with a l2,1-norm regularized logistic regression was the best method for across-subject decoding for both MEG and EEG. In Study II, we also compared the decoding results of simultaneously measured EEG and MEG data, and investigated whether the MEG responses to passive hand movements could be used to train a classifier to detect MI. MEG yielded altogether slightly, but not significantly, better results than EEG. Training the classifiers with subject’s own or other subjects’ passive movements did not result in high accuracy, which indicates that passive movements should not be used for calibrating an MI-BCI. The methods presented in this thesis are suitable for a real-time MEG-based BCI. The decoding results can be used as a benchmark when developing other classifiers specifically for motor imagery -related MEG data.Aivo-tietokone -käyttöliittymät (brain–computer interface; BCI) ovat viime aikoina herättäneet kiinnostusta niin neurotieteen perustutkimuksessa kuin kliinisissä interventioissakin. Suurin osa ei-invasiivisista BCI:stä mittaa aivotoimintaa elektroenkefalografialla (EEG). EEG:n matala signaali-kohinasuhde ja huono avaruudellinen resoluutio kuitenkin hankaloittavat reaaliaikais-ta signaalianalyysia ja aivotoiminnan luokittelua. Nämä rajoitteet voidaan kiertää käyttämällä magnetoenkefalografiaa (MEG) vaihtoehtoisena mittausmenetelmänä. Tämän työn tavoitteena on kehittää käden liikkeen kuvittelua luokitteleva, MEG:hen perustuva BCI, jota voidaan myöhemmin käyttää terapeuttisena menetelmänä esimerkiksi aivoinfarktista toipuvien potilaiden kuntoutuk-sessa. Tutkimuksessa validoidaan terveillä koehenkilöillä tehtyjen MEG-mittausten perusteella koneoppimismenetelmiä, joilla luokitellaan liikkeen kuvittelun aiheuttamaa aivotoimintaa. Ensimmäisessä osatyössä (Tutkimus I) vertailtiin piirteenirrotusmenetelmiä, joita käytetään erottamaan toisistaan vasemman ja oikean käden kuvittelu sekä liikkeen kuvittelu ja lepotila. Ha-vaittiin, että avaruudellisesti suodatettujen signaalien taajuuskaistan teho luokittelupiirteenä tuotti parempia luokittelutarkkuuksia kuin parietaalisista gradiometreistä mitatut aika-taajuuspiirteet. Lisäksi edeltävä avaruudellinen suodatus paransi aika-taajuuspiirteiden erottelukykyä luokittelu-tehtävissä.BCI:n opetusdata kerätään yleensä kunkin mittauskerran alussa. Koska tämä voi kuitenkin olla aikaavievää ja uuvuttavaa koehenkilölle, opetusdatana voidaan käyttää myös muilta koehenkilöiltä kerättyjä mittaussignaaleja. Toisessa osatyössä (Tutkimus II) vertailtiin koehenkilöiden väliseen luo-kitteluun soveltuvia menetelmiä. Tulosten perusteella monitehtäväoppimista ja l2,1-regularisoitua logistista regressiota käyttävä luokittelija oli paras menetelmä koehenkilöiden väliseen luokitteluun sekä MEG:llä että EEG:llä. Toisessa osatyössä vertailtiin myös samanaikaisesti mitattujen MEG:n ja EEG:n tuottamia luokit-telutuloksia, sekä tutkittiin voidaanko passiivisten kädenliikkeiden aikaansaamia MEG-vasteita käyttää liikkeen kuvittelua tunnistavien luokittelijoiden opetukseen. MEG tuotti hieman, muttei merkittävästi, parempia tuloksia kuin EEG. Luokittelijoiden opetus koehenkilöiden omilla tai mui-den koehenkilöiden passiiviliikkeillä ei tuottanut hyviä luokittelutarkkuuksia, mikä osoittaa että passiiviliikkeitä ei tulisi käyttää liikkeen kuvittelua tunnistavan BCI:n kalibrointiin. Työssä esitettyjä menetelmiä voidaan käyttää reaaliaikaisessa MEG-BCI:ssä. Luokittelutuloksia voidaan käyttää vertailukohtana kehitettäessä muita liikkeen kuvitteluun liittyvän MEG-datan luokittelijoita

Mapping & decoding cortical engagement during motor imagery, mental arithmetic, and silent word generation using MEG

Accurate quantification of cortical engagement during mental imagery tasks remains a challenging brain-imaging problem with immediate relevance to developing brain–computer interfaces. We analyzed magnetoencephalography (MEG) data from 18 individuals completing cued motor imagery, mental arithmetic, and silent word generation tasks. Participants imagined movements of both hands (HANDS) and both feet (FEET), subtracted two numbers (SUB), and silently generated words (WORD). The task-related cortical engagement was inferred from beta band (17–25 Hz) power decrements estimated using a frequency-resolved beamforming method. In the hands and feet motor imagery tasks, beta power consistently decreased in premotor and motor areas. In the word and subtraction tasks, beta-power decrements showed engagements in language and arithmetic processing within the temporal, parietal, and inferior frontal regions. A support vector machine classification of beta power decrements yielded high accuracy rates of 74 and 68% for classifying motor-imagery (HANDS vs. FEET) and cognitive (WORD vs. SUB) tasks, respectively. From the motor-versus-nonmotor contrasts, excellent accuracy rates of 85 and 80% were observed for hands-versus-word and hands-versus-sub, respectively. A multivariate Gaussian-process classifier provided an accuracy rate of 60% for the four-way (HANDS-FEET-WORD-SUB) classification problem. Individual task performance was revealed by within-subject correlations of beta-decrements. Beta-power decrements are helpful metrics for mapping and decoding cortical engagement during mental processes in the absence of sensory stimuli or overt behavioral outputs. Markers derived based on beta decrements may be suitable for rehabilitation purposes, to characterize motor or cognitive impairments, or to treat patients recovering from a cerebral stroke

Cortically coupled image computing

In the 1970s, researchers at the University of California started to investigate communication between humans and computers using neural signals, which lead to the emergence of brain- computer interfaces (BCIs). In the past 40 years, significant progress has been achieved in ap- plication areas such as neuroprosthetics and rehabilitation. BCIs have been recently applied to media analytics (e.g., image search and information retrieval) as we are surrounded by tremen- dous amounts of media information today. A cortically coupled computer vision (CCCV) sys- tem is a type of BCI that exposes users to high throughput image streams via the rapid serial visual presentation (RSVP) protocol. Media analytics has also been transformed through the enormous advances in artificial intelligence (AI) in recent times. Understanding and presenting the nature of the human-AI relationship will play an important role in our society in the future. This thesis explores two lines of research in the context of traditional BCIs and AI. Firstly, we study and investigate the fundamental processing methods such as feature extraction and clas- sification for CCCV systems. Secondly, we discuss the feasibility of interfacing neural systems with AI technology through CCCV, an area we identify as neuro-AI interfacing. We have made two electroencephalography (EEG) datasets available to the community that support our inves- tigation of these two research directions. These are the neurally augmented image labelling strategies (NAILS) dataset and the neural indices for face perception analysis (NIFPA) dataset, which are introduced in Chapter 2. The first line of research focuses on studying and investigating fundamental processing methods for CCCV. In Chapter 3, we present a review on recent developments in processing methods for CCCV. This review introduces CCCV related components, specifically the RSVP experimental setup, RSVP-EEG phenomena such as the P300 and N170, evaluation metrics, feature extraction and classification. We then provide a detailed study and an analysis on spatial filtering pipelines in Chapter 4, which are the most widely used feature extraction and reduction methods in a CCCV system. In this context, we propose a spatial filtering technique named multiple time window LDA beamformers (MTWLB) and compare it to two other well-known techniques in the literature, namely xDAWN and common spatial patterns (CSP). Importantly, we demonstrate the efficacy of MTWLB for time-course source signal reconstruction compared to existing methods, which we then use as a source signal information extraction method to support a neuro-AI interface. This will be further discussed in this thesis i.e. Chapter 6 and Chapter 7. The latter part of this thesis investigates the feasibility of neuro-AI interfaces. We present two research studies which contribute to this direction. Firstly, we explore the idea of neuro- AI interfaces based on stimulus and neural systems i.e., observation of the effects of stimuli produced by different AI systems on neural signals. We use generative adversarial networks (GANs) to produce image stimuli in this case as GANs are able to produce higher quality images compared to other deep generative models. Chapter 5 provides a review on GAN-variants in terms of loss functions and architectures. In Chapter 6, we design a comprehensive experiment to verify the effects of images produced by different GANs on participants’ EEG responses. In this we propose a biologically-produced metric called Neuroscore for evaluating GAN per- formance. We highlight the consistency between Neuroscore and human perceptual judgment, which is superior to conventional metrics (i.e., Inception Score (IS), Fre ́chet Inception Distance (FID) and Kernel Maximum Mean Discrepancy (MMD) discussed in this thesis). Secondly, in order to generalize Neuroscore, we explore the use of a neuro-AI interface to help convolutional neural networks (CNNs) predict a Neuroscore with only an image as the input. In this scenario, we feed the reconstructed P300 source signals to the intermediate layer as supervisory informa- tion. We demonstrate that including biological neural information can improve the prediction performance for our proposed CNN models and the predicted Neuroscore is highly correlated with the real Neuroscore (as directly calculated from human neural signals)

Magnetoencephalography

This is a practical book on MEG that covers a wide range of topics. The book begins with a series of reviews on the use of MEG for clinical applications, the study of cognitive functions in various diseases, and one chapter focusing specifically on studies of memory with MEG. There are sections with chapters that describe source localization issues, the use of beamformers and dipole source methods, as well as phase-based analyses, and a step-by-step guide to using dipoles for epilepsy spike analyses. The book ends with a section describing new innovations in MEG systems, namely an on-line real-time MEG data acquisition system, novel applications for MEG research, and a proposal for a helium re-circulation system. With such breadth of topics, there will be a chapter that is of interest to every MEG researcher or clinician

The neurophysiological changes associated with motor learning in adults and adolescents

One main purpose of this dissertation was to explore how sensorimotor cortical oscillations changed after practicing a novel ankle plantarflexion target matching task. We behaviorally quantified the speed, accuracy, reaction time, velocity, and variability of the participant’s performance of the task, while collecting their neurophysiological responses with magnetoencephalography (MEG). With these data, we assessed how the motor planning and execution stages of movement during a goal directed target matching task changed after practicing a task in typically developing young adults with their non-dominant ankle. We found that the cortical oscillations in the beta frequency range that were sourced from the sensorimotor and occipital cortices were weaker after practice. These individuals also improved behaviorally, with faster speed, greater accuracy, higher velocity, and less variability. The decreased strength likely reflects a more refined motor plan, a reduction in neural resources needed to perform the task, and/or an enhancement of the processes that are involved in the visuomotor transformations that occur prior to the onset of the motor action. The second purpose was to explore how the changes of the sensorimotor cortical oscillations after practicing a novel ankle plantarflexion target matching task differ between adults and adolescents. We assessed these behavioral and neurophysiological changes in a cohort of typically developed adults and adolescents. After practice, all of the participants matched more targets, matched the targets faster, had improved accuracy, faster reaction times, and faster force production. However, the motor performance of the adults exceeded what was seen in the adolescents regardless of practice. In conjunction with the behavioral results, the strength of the beta ERD across the motor planning and execution stages was reduced after practice in the sensorimotor cortices of the adolescents, but was stronger in the adults. These outcomes suggest that there are age-dependent changes in the sensorimotor cortical oscillations after practice, which might be related to familiarity with the motor task. The third purpose was to explore how movement attenuates the somatosensory cortical oscillations and how this attenuation differs in adults and adolescents. We used MEG to address this knowledge gap by applying an electrical stimulation to the tibial nerve as adolescents and adults produced an isometric ankle plantarflexion force, or sat quietly with no motor activity. We found movement-related attenuation of the somatosensory oscillations. Attenuation of the alpha-beta ERS while producing the isometric force was greater in adolescents when compared with adults, while the adults had a greater attenuation of the beta ERD. These results imply that alterations of frequency specific somatosensory cortical oscillations may partly underlie the altered motor performance characteristics seen in adolescents