Sensorimotor rhythm brain-computer interface – A game-based online co-adaptive training

Tese de mestrado integrado, Engenharia Biomédica e Biofísica (Engenharia Clínica e Instrumentação Médica) Universidade de Lisboa, Faculdade de Ciências, 2018Brain-Computer Interface (BCI) technology translates brain signals into messages. BCI users are thus enabled to interact with the environment by thought, or more generally speaking by mental processes. Event-related desynchronization (ERD) based BCIs use the detection of changes in the spontaneous electroencephalogram (EEG) signal. Different mental processes induce power decreases (ERD) or increases (event-related synchronization, ERS) in different frequencies and different areas of the brain. These differences can be measured and classified. Operating a non-invasive EEG based sensorimotor rhythm BCI is a skill that typically requires extensive training. Lately, online co-adaptive feedback training approaches achieved promising results after short periods of training. Does this also mean that users can have meaningful BCI-based interactions after training, when the BCI is no longer adapting, like in a real- life scenario? To answer this question an online study was conducted with 20 naïve (first time) users. After a short (less than 20 minutes) setup, the users trained to gain BCI control by playing a Whack- A-Mole game where they would have to perform Motor Imagery (imagination of a specific movement- MI) to control a hammer to hit a mole. The game was played for about 30 minutes. During this time, the user learns to perform MI with online feedback from the game and the BCI parameters recurrently adapt to the user’s EEG patterns every~1minute. This recurrent adaptation allows different users to use slightly different strategies and produce ERDs in different frequencies and brain areas without loss of performance. After 30 minutes of training the adaptation was stopped and the users continued playing the game with the trained BCI for another 20 minutes. The BCI parameters were calibrated with data from the adaptive stage and kept fixed in the last 20 minutes. Our hypothesis is that once a system was co-adaptively trained it can maintain its performance without recurrent adaptation. Eighteen out of the twenty users were able to control the BCI and play the game. Seventeen out of the eighteen were able to improve or keep performance between adaptive and non-adaptive stage. These results seem to suggest that online co-adaptation is an effective way to gain BCI control

MEG:hen perustuvan aivo-tietokone -käyttöliittymän kehitys

Brain–computer interfaces (BCI) have recently gained interest both in basic neuroscience and clinical interventions. The majority of noninvasive BCIs measure brain activity with electroencephalography (EEG). However, the real-time signal analysis and decoding of brain activity suffer from low signal-to-noise ratio and poor spatial resolution of EEG. These limitations could be overcome by using magnetoencephalography (MEG) as an alternative measurement modality. The aim of this thesis is to develop an MEG-based BCI for decoding hand motor imagery, which could eventually serve as a therapeutic method for patients recovering from e.g. cerebral stroke. Here, machine learning methods for decoding motor imagery -related brain activity are validated with healthy subjects’ MEG measurements. The first part of the thesis (Study I) involves a comparison of feature extraction methods for classifying left- vs right-hand motor imagery (MI), and MI vs rest. It was found that spatial filtering and further extraction of bandpower features yield better classification accuracy than time–frequency features extracted from parietal gradiometers. Furthermore, prior spatial filtering improved the discrimination capability of time–frequency features. The training data for a BCI is typically collected in the beginning of each measurement session. However, as this can be time-consuming and exhausting for the subject, the training data from other subjects’ measurements could be used as well. In the second part of the thesis (Study II), methods for across-subject classification of MI were compared. The results showed that a classifier based on multi-task learning with a l2,1-norm regularized logistic regression was the best method for across-subject decoding for both MEG and EEG. In Study II, we also compared the decoding results of simultaneously measured EEG and MEG data, and investigated whether the MEG responses to passive hand movements could be used to train a classifier to detect MI. MEG yielded altogether slightly, but not significantly, better results than EEG. Training the classifiers with subject’s own or other subjects’ passive movements did not result in high accuracy, which indicates that passive movements should not be used for calibrating an MI-BCI. The methods presented in this thesis are suitable for a real-time MEG-based BCI. The decoding results can be used as a benchmark when developing other classifiers specifically for motor imagery -related MEG data.Aivo-tietokone -käyttöliittymät (brain–computer interface; BCI) ovat viime aikoina herättäneet kiinnostusta niin neurotieteen perustutkimuksessa kuin kliinisissä interventioissakin. Suurin osa ei-invasiivisista BCI:stä mittaa aivotoimintaa elektroenkefalografialla (EEG). EEG:n matala signaali-kohinasuhde ja huono avaruudellinen resoluutio kuitenkin hankaloittavat reaaliaikais-ta signaalianalyysia ja aivotoiminnan luokittelua. Nämä rajoitteet voidaan kiertää käyttämällä magnetoenkefalografiaa (MEG) vaihtoehtoisena mittausmenetelmänä. Tämän työn tavoitteena on kehittää käden liikkeen kuvittelua luokitteleva, MEG:hen perustuva BCI, jota voidaan myöhemmin käyttää terapeuttisena menetelmänä esimerkiksi aivoinfarktista toipuvien potilaiden kuntoutuk-sessa. Tutkimuksessa validoidaan terveillä koehenkilöillä tehtyjen MEG-mittausten perusteella koneoppimismenetelmiä, joilla luokitellaan liikkeen kuvittelun aiheuttamaa aivotoimintaa. Ensimmäisessä osatyössä (Tutkimus I) vertailtiin piirteenirrotusmenetelmiä, joita käytetään erottamaan toisistaan vasemman ja oikean käden kuvittelu sekä liikkeen kuvittelu ja lepotila. Ha-vaittiin, että avaruudellisesti suodatettujen signaalien taajuuskaistan teho luokittelupiirteenä tuotti parempia luokittelutarkkuuksia kuin parietaalisista gradiometreistä mitatut aika-taajuuspiirteet. Lisäksi edeltävä avaruudellinen suodatus paransi aika-taajuuspiirteiden erottelukykyä luokittelu-tehtävissä.BCI:n opetusdata kerätään yleensä kunkin mittauskerran alussa. Koska tämä voi kuitenkin olla aikaavievää ja uuvuttavaa koehenkilölle, opetusdatana voidaan käyttää myös muilta koehenkilöiltä kerättyjä mittaussignaaleja. Toisessa osatyössä (Tutkimus II) vertailtiin koehenkilöiden väliseen luo-kitteluun soveltuvia menetelmiä. Tulosten perusteella monitehtäväoppimista ja l2,1-regularisoitua logistista regressiota käyttävä luokittelija oli paras menetelmä koehenkilöiden väliseen luokitteluun sekä MEG:llä että EEG:llä. Toisessa osatyössä vertailtiin myös samanaikaisesti mitattujen MEG:n ja EEG:n tuottamia luokit-telutuloksia, sekä tutkittiin voidaanko passiivisten kädenliikkeiden aikaansaamia MEG-vasteita käyttää liikkeen kuvittelua tunnistavien luokittelijoiden opetukseen. MEG tuotti hieman, muttei merkittävästi, parempia tuloksia kuin EEG. Luokittelijoiden opetus koehenkilöiden omilla tai mui-den koehenkilöiden passiiviliikkeillä ei tuottanut hyviä luokittelutarkkuuksia, mikä osoittaa että passiiviliikkeitä ei tulisi käyttää liikkeen kuvittelua tunnistavan BCI:n kalibrointiin. Työssä esitettyjä menetelmiä voidaan käyttää reaaliaikaisessa MEG-BCI:ssä. Luokittelutuloksia voidaan käyttää vertailukohtana kehitettäessä muita liikkeen kuvitteluun liittyvän MEG-datan luokittelijoita

Challenges and opportunities for the future of Brain-Computer Interface in neurorehabilitation

Brain-computer interfaces (BCIs) provide a unique technological solution to circumvent the damaged motor system. For neurorehabilitation, the BCI can be used to translate neural signals associated with movement intentions into tangible feedback for the patient, when they are unable to generate functional movement themselves. Clinical interest in BCI is growing rapidly, as it would facilitate rehabilitation to commence earlier following brain damage and provides options for patients who are unable to partake in traditional physical therapy. However, substantial challenges with existing BCI implementations have prevented its widespread adoption. Recent advances in knowledge and technology provide opportunities to facilitate a change, provided that researchers and clinicians using BCI agree on standardisation of guidelines for protocols and shared efforts to uncover mechanisms. We propose that addressing the speed and effectiveness of learning BCI control are priorities for the field, which may be improved by multimodal or multi-stage approaches harnessing more sensitive neuroimaging technologies in the early learning stages, before transitioning to more practical, mobile implementations. Clarification of the neural mechanisms that give rise to improvement in motor function is an essential next step towards justifying clinical use of BCI. In particular, quantifying the unknown contribution of non-motor mechanisms to motor recovery calls for more stringent control conditions in experimental work. Here we provide a contemporary viewpoint on the factors impeding the scalability of BCI. Further, we provide a future outlook for optimal design of the technology to best exploit its unique potential, and best practices for research and reporting of findings

The Hybrid BCI

Nowadays, everybody knows what a hybrid car is. A hybrid car normally has two engines to enhance energy efficiency and reduce CO2 output. Similarly, a hybrid brain-computer interface (BCI) is composed of two BCIs, or at least one BCI and another system. A hybrid BCI, like any BCI, must fulfill the following four criteria: (i) the device must rely on signals recorded directly from the brain; (ii) there must be at least one recordable brain signal that the user can intentionally modulate to effect goal-directed behaviour; (iii) real time processing; and (iv) the user must obtain feedback. This paper introduces hybrid BCIs that have already been published or are in development. We also introduce concepts for future work. We describe BCIs that classify two EEG patterns: one is the event-related (de)synchronisation (ERD, ERS) of sensorimotor rhythms, and the other is the steady-state visual evoked potential (SSVEP). Hybrid BCIs can either process their inputs simultaneously, or operate two systems sequentially, where the first system can act as a “brain switch”. For example, we describe a hybrid BCI that simultaneously combines ERD and SSVEP BCIs. We also describe a sequential hybrid BCI, in which subjects could use a brain switch to control an SSVEP-based hand orthosis. Subjects who used this hybrid BCI exhibited about half the false positives encountered while using the SSVEP BCI alone. A brain switch can also rely on hemodynamic changes measured through near-infrared spectroscopy (NIRS). Hybrid BCIs can also use one brain signal and a different type of input. This additional input can be an electrophysiological signal such as the heart rate, or a signal from an external device such as an eye tracking system

Does Fractional Anisotropy Predict Motor Imagery Neurofeedback Performance in Healthy Older Adults?

Motor imagery neurofeedback training has been proposed as a potential add-on therapy for motor impairment after stroke, but not everyone benefits from it. Previous work has used white matter integrity to predict motor imagery neurofeedback aptitude in healthy young adults. We set out to test this approach with motor imagery neurofeedback that is closer to that used for stroke rehabilitation and in a sample whose age is closer to that of typical stroke patients. Using shrinkage linear discriminant analysis with fractional anisotropy values in 48 white matter regions as predictors, we predicted whether each participant in a sample of 21 healthy older adults (48–77 years old) was a good or a bad performer with 84.8% accuracy. However, the regions used for prediction in our sample differed from those identified previously, and previously suggested regions did not yield significant prediction in our sample. Including demographic and cognitive variables which may correlate with motor imagery neurofeedback performance and white matter structure as candidate predictors revealed an association with age but also led to loss of statistical significance and somewhat poorer prediction accuracy (69.6%). Our results suggest cast doubt on the feasibility of predicting the benefit of motor imagery neurofeedback from fractional anisotropy. At the very least, such predictions should be based on data collected using the same paradigm and with subjects whose characteristics match those of the target case as closely as possible

BRAIN-COMPUTER MUSIC INTERFACING: DESIGNING PRACTICAL SYSTEMS FOR CREATIVE APPLICATIONS

Brain-computer music interfacing (BCMI) presents a novel approach to music making, as it requires only the brainwaves of a user to control musical parameters. This presents immediate benefits for users with motor disabilities that may otherwise prevent them from engaging in traditional musical activities such as composition, performance or collaboration with other musicians. BCMI systems with active control, where a user can make cognitive choices that are detected within brain signals, provide a platform for developing new approaches towards accomplishing these activities. BCMI systems that use passive control present an interesting alternate to active control, where control over music is accomplished by harnessing brainwave patterns that are associated with subconscious mental states. Recent developments in brainwave measuring technologies, in particular electroencephalography (EEG), have made brainwave interaction with computer systems more affordable and accessible and the time is ripe for research into the potential such technologies can offer for creative applications for users of all abilities. This thesis presents an account of BCMI development that investigates methods of active, passive and hybrid (multiple control methods) control that include control over electronic music, acoustic instrumental music, multi-brain systems and combining methods of brainwave control. In practice there are many obstacles associated with detecting useful brainwave signals, in particular when scaling systems otherwise designed for medical studies for use outside of laboratory settings. Two key areas are addressed throughout this thesis. Firstly, improving the accuracy of meaningful brain signal detection in BCMI, and secondly, exploring the creativity available in user control through ways in which brainwaves can be mapped to musical features. Six BCMIs are presented in this thesis, each with the objective of exploring a unique aspect of user control. Four of these systems are designed for live BCMI concert performance, one evaluates a proof-of-concept through end-user testing and one is designed as a musical composition tool. The thesis begins by exploring the field of brainwave detection and control and identifies the steady-state visually evoked potential (SSVEP) method of eliciting brainwave control as a suitable technique for use in BCMI. In an attempt to improve signal accuracy of the SSVEP technique a new modular hardware unit is presented that provides accurate SSVEP stimuli, suitable for live music performance. Experimental data confirms the performance of the unit in tests across three different EEG hardware platforms. Results across 11 users indicate that a mean accuracy of 96% and an average response time of 3.88 seconds are attainable with the system. These results contribute to the development of the BCMI for Activating Memory, a multi-user system. Once a stable SSVEP platform is developed, control is extended through the integration of two more brainwave control techniques: affective (emotional) state detection and motor imagery response. In order to ascertain the suitability of the former an experiment confirms the accuracy of EEG when measuring affective states in response to music in a pilot study. This thesis demonstrates how a range of brainwave detection methods can be used for creative control in musical applications. Video and audio excerpts of BCMI pieces are also included in the Appendices

Lightweight Machine Learning with Brain Signals

Electroencephalography(EEG) signals are gaining popularity in Brain-Computer Interface(BCI) systems and neural engineering applications thanks to their portability and availability. Inevitably, the sensory electrodes on the entire scalp would collect signals irrelevant to the particular BCI task, increasing the risks of overfitting in machine learning-based predictions. While this issue is being addressed by scaling up the EEG datasets and handcrafting the complex predictive models, this also leads to increased computation costs. Moreover, the model trained for one set of subjects cannot easily be adapted to other sets due to inter-subject variability, which creates even higher over-fitting risks. Meanwhile, despite previous studies using either convolutional neural networks(CNNs) or graph neural networks(GNNs) to determine spatial correlations between brain regions, they fail to capture brain functional connectivity beyond physical proximity. To this end, we propose 1) removing task-irrelevant noises instead of merely complicating models; 2) extracting subject-invariant discriminative EEG encodings, by taking functional connectivity into account; 3) navigating and training deep learning model with the most critical EEG channels; 4) detecting most similar EEG segments with target subject to reduce the cost of computation as well as inter-subject variability. Specifically, we construct a task-adaptive graph representation of brain network based on topological functional connectivity rather than distance-based connections. Further, non-contributory EEG channels are excluded by selecting only functional regions relevant to the corresponding intention. Lastly, contributory EEG segments are detected by several similarity estimation metrics, we then evaluate and train our proposed framework upon detected EEG segments to compare the performance of different metrics in EEG BCI tasks. We empirically show that our proposed approach, SIFT-EEG, outperforms state-of-the-art, with around 4% and 7% improvements over CNN-based and GNN-based models, on performing motor imagery predictions. Also, the task-adaptive channel selection demonstrates similar predictive performance with only 20% of raw EEG data. Moreover, the best-performed metric can achieve a high level of accuracy with less than 9% training data, suggesting a possible shift in direction for future works other than simply scaling up the model

Development of A Versatile Multichannel CWNIRS Instrument for Optical Brain-Computer Interface Applications

This thesis describes the design, development, and implementation of a versatile multichannel continuous-wave near-infrared spectroscopy (CWNIRS) instrument for brain-computer interface (BCI) applications. Specifically, it was of interest to assess what gains could be achieved by using a multichannel device compared to the single channel device implemented by Coyle in 2004. Moreover, the multichannel approach allows for the assessment of localisation of functional tasks in the cerebral cortex, and can identify lateralisation of haemodynamic responses to motor events. The approach taken to extend single channel to multichannel was based on a software-controlled interface. This interface allowed flexibility in the control of individual optodes including their synchronisation and modulation (AM, TDM, CDMA). Furthermore, an LED driver was developed for custom-made triple-wavelength LEDs. The system was commissioned using a series of experiments to verify the performance of individual components in the system. The system was then used to carry out a set of functional studies including motor imagery and cognitive tasks. The experimental protocols based on motor imagery and overt motor tasks were verified by comparison with fMRI. The multichannel approach identified stroke rehabilitation as a new application area for optical BCI. In addition, concentration changes in deoxyhaemoglobin were identified as being a more localised indicator of functional activity, which is important for effective BCI design. An assessment was made on the effect of the duration of the stimulus period on the haemodynamic signals. This demonstrated the possible benefits of using a shorter stimulus period to reduce the adverse affects of low blood pressure oscillations. i