Brain Computer Interfaces for the Control of Robotic Swarms

abstract: A robotic swarm can be defined as a large group of inexpensive, interchangeable robots with limited sensing and/or actuating capabilities that cooperate (explicitly or implicitly) based on local communications and sensing in order to complete a mission. Its inherent redundancy provides flexibility and robustness to failures and environmental disturbances which guarantee the proper completion of the required task. At the same time, human intuition and cognition can prove very useful in extreme situations where a fast and reliable solution is needed. This idea led to the creation of the field of Human-Swarm Interfaces (HSI) which attempts to incorporate the human element into the control of robotic swarms for increased robustness and reliability. The aim of the present work is to extend the current state-of-the-art in HSI by applying ideas and principles from the field of Brain-Computer Interfaces (BCI), which has proven to be very useful for people with motor disabilities. At first, a preliminary investigation about the connection of brain activity and the observation of swarm collective behaviors is conducted. After showing that such a connection may exist, a hybrid BCI system is presented for the control of a swarm of quadrotors. The system is based on the combination of motor imagery and the input from a game controller, while its feasibility is proven through an extensive experimental process. Finally, speech imagery is proposed as an alternative mental task for BCI applications. This is done through a series of rigorous experiments and appropriate data analysis. This work suggests that the integration of BCI principles in HSI applications can be successful and it can potentially lead to systems that are more intuitive for the users than the current state-of-the-art. At the same time, it motivates further research in the area and sets the stepping stones for the potential development of the field of Brain-Swarm Interfaces (BSI).Dissertation/ThesisMasters Thesis Mechanical Engineering 201

Review of the BCI Competition IV

The BCI competition IV stands in the tradition of prior BCI competitions that aim to provide high quality neuroscientific data for open access to the scientific community. As experienced already in prior competitions not only scientists from the narrow field of BCI compete, but scholars with a broad variety of backgrounds and nationalities. They include high specialists as well as students. The goals of all BCI competitions have always been to challenge with respect to novel paradigms and complex data. We report on the following challenges: (1) asynchronous data, (2) synthetic, (3) multi-class continuous data, (4) session-to-session transfer, (5) directionally modulated MEG, (6) finger movements recorded by ECoG. As after past competitions, our hope is that winning entries may enhance the analysis methods of future BCIs.BMBF, 01IB001A, LOKI - Lernen zur Organisation komplexer Systeme der Informationsverarbeitung - Lernen im Kontext der SzenenanalyseBMBF, 01GQ0850, Bernstein Fokus Neurotechnologie - Nichtinvasive Neurotechnologie für Mensch-Maschine InteraktionEC/FP7/224631/EU/Tools for Brain-Computer Interaction/TOBIEC/FP7/216886/EU/Pattern Analysis, Statistical Modelling and Computational Learning 2/PASCAL2BMBF, 01GQ0420, Verbundprojekt: Bernstein-Zentrum für Neural Dynamics, Freiburg - CNDFBMBF, 01GQ0761, Bewegungsassoziierte Aktivierung - Dekodierung bewegungsassoziierter GehirnsignaleBMBF, 01GQ0762, Bewegungsassoziierte Aktivierung - Gehirn- und Maschinenlerne

Classification of Frequency and Phase Encoded Steady State Visual Evoked Potentials for Brain Computer Interface Speller Applications using Convolutional Neural Networks

Over the past decade there have been substantial improvements in vision based Brain-Computer Interface (BCI) spellers for quadriplegic patient populations. This thesis contains a review of the numerous bio-signals available to BCI researchers, as well as a brief chronology of foremost decoding methodologies used to date. Recent advances in classification accuracy and information transfer rate can be primarily attributed to time consuming patient specific parameter optimization procedures. The aim of the current study was to develop analysis software with potential ‘plug-in-and-play’ functionality. To this end, convolutional neural networks, presently established as state of the art analytical techniques for image processing, were utilized. The thesis herein defines deep convolutional neural network architecture for the offline classification of phase and frequency encoded SSVEP bio-signals. Networks were trained using an extensive 35 participant open source Electroencephalographic (EEG) benchmark dataset (Department of Bio-medical Engineering, Tsinghua University, Beijing). Average classification accuracies of 82.24% and information transfer rates of 22.22 bpm were achieved on a BCI naïve participant dataset for a 40 target alphanumeric display, in absence of any patient specific parameter optimization

Study of Adaptation Methods Towards Advanced Brain-computer Interfaces

Ph.DDOCTOR OF PHILOSOPH

Under-sampling and Classification of P300 Single-Trials using Self-Organized Maps and Deep Neural Networks for a Speller BCI

A Brain-Computer Interface (BCI) allows its userto control machines or other devices by translating its brainactivity and using it as commands. This kind of technologyhas as potential users people with motor disabilities since itwould allow them to interact with their environment withoutusing their peripheral nerves, helping them to regain their lostautonomy. One of the most successful BCI applications is theP300-based Speller. Its operation depends entirely on its capacityto identify and discriminate the presence of the P300 potentialsfrom electroencephalographic (EEG) signals. For the system to dothis correctly, it is necessary to choose an adequate classifier andtrain it with a balanced data-set. However, due to the use of anoddball paradigm to elicit the P300 potential, only unbalanceddata-sets can be obtained. This paper focuses on the trainingstage of two classifiers, a deep feedforward network (DFN) anda deep belief network (DBN), to be used in a P300-based BCI. Thedata-sets obtained from healthy subjects and post-stroke victimswere pre-processed and then balanced using a Self-OrganizingMaps-based under-sampling approach prior training looking toincrease the accuracy of the classifiers. We compared the resultswith our previous works and observed an increase of 7% inclassification accuracy for the most critical subject. The DFNachieved a maximum classification accuracy of 93.29% for apost-stroke subject and 93.60% for a healthy one

Brain computer interfaces: an engineering view. Design, implementation and test of a SSVEP-based BCI.

This thesis presents the realization of a compact, yet flexible BCI platform, which, when compared to most commercially-available solution, can offer an optimal trade-off between the following requirements: (i) minimal, easy experimental setup; (ii) flexibility, allowing simultaneous studies on other bio-potentials; (iii) cost effectiveness (e.g. < 1000 €); (iv) robust design, suitable for operation outside lab environments. The thesis encompasses all the project phases, from hardware design and realization, up to software and signal processing. The work started from the development of the hardware acquisition unit. It resulted in a compact, battery-operated module, whose medium-to-large scale production costs are in the range of 300 €. The module features 16 input channels and can be used to acquire different bio-potentials, including EEG, EMG, ECG. Module performance is very good (RTI noise < 1.3 uVpp), and was favourably compared against a commercial device (g.tec USBamp). The device was integrated into an ad-hoc developed Matlab-based platform, which handles the hardware control, as well as the data streaming, logging and processing. Via a specifically developed plug-in, incoming data can also be streamed to a TOBI-interface compatible system. As a demonstrator, the BCI was developed for AAL (Ambient Assisted Living) system-control purposes, having in mind the following requirements: (i) online, self-paced BCI operation (i.e., the BCI monitors the EEG in real-time and must discern between intentional control periods, and non-intentional, rest ones, interpreting the user’s intent only in the first case); (ii) calibration-free approach (“ready-to-use”, “Plug&Play”); (iii) subject-independence (general approach). The choice of the BCI operating paradigm fell on Steady State visual Evoked Potential (SSVEP). Two offline SSVEP classification algorithms were proposed and compared against reference literature, highlighting good performance, especially in terms of lower computational complexity. A method for improving classification accuracy was presented, suitable for use in online, self-paced scenarios (since it can be used to discriminate between intentional control periods and non-intentional ones). Results show a very good performance, in particular in terms of false positives immunity (0.26 min^-1), significantly improving over the state of the art. The whole BCI setup was tested both in lab condition, as well as in relatively harsher ones (in terms of environmental noise and non-idealities), such as in the context of the Handimatica 2014 exhibition. In both cases, a demonstrator allowing control of home appliances through BCI was developed

SaS-BCI: A New Strategy to Predict Image Memorability and use Mental Imagery as a Brain-Based Biometric Authentication

Security authentication is one of the most important levels of information security. Nowadays, human biometric techniques are the most secure methods for authentication purposes that cover the problems of older types of authentication like passwords and pins. There are many advantages of recent biometrics in terms of security; however, they still have some disadvantages. Progresses in technology made some specific devices, which make it possible to copy and make a fake human biometric because they are all visible and touchable. According to this matter, there is a need for a new biometric to cover the issues of other types. Brainwave is human data, which uses them as a new type of security authentication that has engaged many researchers. There are some research and experiments, which are investigating and testing EEG signals to find the uniqueness of human brainwave. Some researchers achieved high accuracy rates in this area by applying different signal acquisition techniques, feature extraction and classifications using Brain–Computer Interface (BCI). One of the important parts of any BCI processes is the way that brainwaves could be acquired and recorded. A new Signal Acquisition Strategy is presented in this paper for the process of authorization and authentication of brain signals specifically. This is to predict image memorability from the user’s brain to use mental imagery as a visualization pattern for security authentication. Therefore, users can authenticate themselves with visualizing a specific picture in their minds. In conclusion, we can see that brainwaves can be different according to the mental tasks, which it would make it harder using them for authentication process. There are many signal acquisition strategies and signal processing for brain-based authentication that by using the right methods, a higher level of accuracy rate could be achieved which is suitable for using brain signal as another biometric security authentication

Cortically coupled image computing

In the 1970s, researchers at the University of California started to investigate communication between humans and computers using neural signals, which lead to the emergence of brain- computer interfaces (BCIs). In the past 40 years, significant progress has been achieved in ap- plication areas such as neuroprosthetics and rehabilitation. BCIs have been recently applied to media analytics (e.g., image search and information retrieval) as we are surrounded by tremen- dous amounts of media information today. A cortically coupled computer vision (CCCV) sys- tem is a type of BCI that exposes users to high throughput image streams via the rapid serial visual presentation (RSVP) protocol. Media analytics has also been transformed through the enormous advances in artificial intelligence (AI) in recent times. Understanding and presenting the nature of the human-AI relationship will play an important role in our society in the future. This thesis explores two lines of research in the context of traditional BCIs and AI. Firstly, we study and investigate the fundamental processing methods such as feature extraction and clas- sification for CCCV systems. Secondly, we discuss the feasibility of interfacing neural systems with AI technology through CCCV, an area we identify as neuro-AI interfacing. We have made two electroencephalography (EEG) datasets available to the community that support our inves- tigation of these two research directions. These are the neurally augmented image labelling strategies (NAILS) dataset and the neural indices for face perception analysis (NIFPA) dataset, which are introduced in Chapter 2. The first line of research focuses on studying and investigating fundamental processing methods for CCCV. In Chapter 3, we present a review on recent developments in processing methods for CCCV. This review introduces CCCV related components, specifically the RSVP experimental setup, RSVP-EEG phenomena such as the P300 and N170, evaluation metrics, feature extraction and classification. We then provide a detailed study and an analysis on spatial filtering pipelines in Chapter 4, which are the most widely used feature extraction and reduction methods in a CCCV system. In this context, we propose a spatial filtering technique named multiple time window LDA beamformers (MTWLB) and compare it to two other well-known techniques in the literature, namely xDAWN and common spatial patterns (CSP). Importantly, we demonstrate the efficacy of MTWLB for time-course source signal reconstruction compared to existing methods, which we then use as a source signal information extraction method to support a neuro-AI interface. This will be further discussed in this thesis i.e. Chapter 6 and Chapter 7. The latter part of this thesis investigates the feasibility of neuro-AI interfaces. We present two research studies which contribute to this direction. Firstly, we explore the idea of neuro- AI interfaces based on stimulus and neural systems i.e., observation of the effects of stimuli produced by different AI systems on neural signals. We use generative adversarial networks (GANs) to produce image stimuli in this case as GANs are able to produce higher quality images compared to other deep generative models. Chapter 5 provides a review on GAN-variants in terms of loss functions and architectures. In Chapter 6, we design a comprehensive experiment to verify the effects of images produced by different GANs on participants’ EEG responses. In this we propose a biologically-produced metric called Neuroscore for evaluating GAN per- formance. We highlight the consistency between Neuroscore and human perceptual judgment, which is superior to conventional metrics (i.e., Inception Score (IS), Fre ́chet Inception Distance (FID) and Kernel Maximum Mean Discrepancy (MMD) discussed in this thesis). Secondly, in order to generalize Neuroscore, we explore the use of a neuro-AI interface to help convolutional neural networks (CNNs) predict a Neuroscore with only an image as the input. In this scenario, we feed the reconstructed P300 source signals to the intermediate layer as supervisory informa- tion. We demonstrate that including biological neural information can improve the prediction performance for our proposed CNN models and the predicted Neuroscore is highly correlated with the real Neuroscore (as directly calculated from human neural signals)