Systems using brain-generated signals can control complex, smart devices by taking into account information about the situation at hand, as well as the operator’s cognitive state

Chavarriaga, Ricardo

Millán, José del R.

Infoscience - École polytechnique fédérale de Lausanne

10.2417/2201006.003009Context-aware brain-computerinterfacesRicardo Chavarriaga and Jose´ del R. Milla´nSystems using brain-generated signals can control complex, smart de-vices by taking into account information about the situation at hand,as well as the operator’s cognitive state.Developments in neuroscience, signal processing and machinelearning are enabling device control using our brain activity.Specifically, current technology allows us to record brain-generated signals in real time, infer the human intention andtranslate it into control commands for external devices. Thesebrain-computer interfaces (BCIs) can currently control virtualkeyboards, games, smart wheelchairs and mobile robots.1 Mostcommonly, BCI systems are based on electrical brain activ-ity recorded on the scalp (using electroencephalograms: EEGs).Machine-learning techniques help to classify these signals intopre-defined patterns of activity associated with particular inten-tions (e.g., imagining moving one’s left hand will lead to devicemotion toward the left). BCI applications have traditionallyfocussed on subjects suffering frommotor handicaps (caused by,for instance, spinal-cord injury, degenerative diseases or locked-in syndrome). Their predominant aim has been restoration orsubstitution of communication and/or motor capabilities, al-though recent developments have also explored their use inhealthy subjects in applications ranging from games2 to imagebrowsing3 and space-system applications.4However, despite their impressive achievements, BCI applica-tions are strongly limited by their low throughput and the smallnumber of commands they can deliver. Designing context-awareinterfaces has been proposed as a way to cope with these limi-tations. Using this approach, the interface collects informationabout the state of the device, as well as its environment, andcombines this with the commands it has decoded from brain ac-tivity (see Figure 1). This enables the performance of complextasks with a reduced number of mental commands (typicallytwo or three) and using the latter to signal high-level instruc-tions while smart devices take care of low-level controlsignals.For instance, we have shown that noninvasive BCIs can beused for real-time control of an intelligent wheelchair in real-istic conditions.5 In this application, BCI commands are limitedto general directions of movement (i.e., move forward, turn leftFigure 1. Context-aware brain-computer interface (BCI). The tra-ditional BCI control loop is enriched by the addition of contextualinformation describing the environment and the user’s cognitive state.EEG: Electroencephalogram.or right), which are interpreted by the wheelchair in conjunc-tion with information from the on-board sensors to computethe actual control commands needed for execution (i.e., speedand angle of movement) to travel smooth trajectories and avoidobstacles. Alternative approaches in context-aware BCI roboticsapplications dynamically change the behaviour correspondingto a particular mental task depending on context. For instance,when controlling a mobile robot, a ‘left’ command signalled bythe BCI would have different meaning depending on whetheror not there is a wall on that side of the robot. The robot will ei-ther move along the wall or turn to the left on the spot.6 Eitherway, this shared-control approach increases the robustness of theoverall system, allowing it to perform complex tasks.The interface can also extract information about the subject’scognitive and perceptual state from the recorded brain activity,Continued on next page10.2417/2201006.003009 Page 2/3Figure 2. EEG error-based navigation.9 (top) The robot movesautonomously using sensory information about its surroundings.(bottom) Wherever it cannot take a reliable decision, it proposes anaction using visual feedback. Error-related signals are decoded from theelicited EEG activity to confirm or reject this proposal.such as awareness of erroneous decisions, either taken by thesubject itself or by an external agent. We have shown that it ispossible to detect error-related EEG activity in real time in sin-gle trials and use this as corrective or learning signals for BCIsystems.7, 8 In addition, in the framework of semi-autonomousnavigation (similar to our wheelchair experiment), we tested atele-operated robot platform that navigates autonomously in in-door environments using its on-board sensors until it reaches adecision point (because it does not know the target destination).At this location, it uses visual feedback to propose a possibleaction (see Figure 2). That action is either selected or discardedbased on online detection of error-related EEG potentials. A userremotely commands the robot while observing a video streamprovided by an on-board camera. The visual feedback is su-perimposed onto the video image.9 Online experiments on onesubject using both real and simulated robots show that it ispossible to successfully guide the robot while providing a nat-ural approach to brain-machine interaction that reduces theuser’s cognitive load (the system behaves autonomously 82% ofthe time).Error-related EEG signals can also be used to adapt theinterface’s behaviour. We recently explored whether similarpotentials can be used to assess and improve the system’s per-formance. We designed a hybrid approach for human-computerinteraction that uses human gestures to send commands to aFigure 3. Self-adaptation in human-computer interaction.10 The com-puter game is controlled by motion-based gesture recognition whileEEG-decoded error-related activity is used for self-adaptation. EMG:Electromyographic. EOG: Electrooculographic.computer and exploits brain activity to provide implicit feed-back about the degree of recognition of such commands (seeFigure 3). Using a simple computer game controlled bywearablemotion sensors, we showed that EEG activity evoked by erro-neous gesture recognition can be classified in single trials at wellabove random levels. Thus, the gesture-recognition system be-comes self-aware of its performance.10 Moreover, we designeda simple adaptation mechanism which uses the EEG signal tolabel newly acquired samples that can be used to recalibrate thegesture-recognition system in a self-supervised fashion. Offlineanalysis shows that this technique can significantly improve theaccuracy of independent gesture recognition for most subjectstested.In summary, the field of BCI has experienced astonishingrecent developments, highlighting the feasibility of using brainactivity to efficiently control complex devices despite existinglimitations in terms of reliability and number of commands.We have shown that enriching the interface with contextualinformation yields more robust results. Robotic applications cancollect information about their environments by employing on-board sensors and use this information to better interpret theuser’s intentions based on the BCI, thus allowing the user anddevice to share the responsibility of control. EEG signals canalso provide information about the subject’s assessment of thedevice’s performance. In particular, error-related activity can beContinued on next page10.2417/2201006.003009 Page 3/3used to identify erroneous system decisions. This informationcan then be used to trigger corrective actions or adapt thesystem in a self-supervised manner.Our future research efforts include studies of mechanisms tochange the level of the system’s autonomy, depending on thecontext or expected reliability (e.g., giving more or less respon-sibility to the system depending on whether the environment iswell known). Other mental states decoded from brain activity(such as fatigue, attention or alarm) can also be included tobetter meet the user’s needs.This work has been supported by the European Union’s Future andEmerging Technologies project ICT-225938 Opportunity: Activity andContext Recognition with Opportunistic Sensor Configuration, Infor-mation and Communication Technologies project FP7-224631 TOBI:Tools for Brain-Computer Interaction and Integrated Project FP6-IST-027140 BACS: Bayesian Approaches to Cognitive Systems. This paperonly reflects the authors’ views and funding agencies are not liable forany use that may be made of the information contained herein.Author InformationRicardo Chavarriaga and Jose´ del R. Milla´nEcole Polytechnique Fe´de´rale de LausanneLausanne, SwitzerlandRicardo Chavarriaga is a research scientist. He focuses on analy-sis of brain electrical signals and designs brain-computer inter-faces. In particular, he is interested in studying human cognitiveprocessing and its potential use in human-machine interactions.References1. J. del R. Milla´n, P. W. Ferrez, F. Gala´n, E. Lew, and R. Chavarriaga,Non-invasivebrain-machine interaction, Int’l J. Pattern Recognit. Artif. Intell. 22, pp. 959–972,2008.2. M. W. Tangermann, M. Krauledat, K. Grzeska, M. Sagebaum, C. Vidaurre,B. Blankertz, and K. R. Mu¨ller, Playing pinball with non-invasive BCI, Adv.Neural Inf. Process. Syst. 21, pp. 1641–1648, 2008.3. A. D. Gerson, L. C. Parra, and P. Sajda, Cortically-coupled computer vision for rapidimage search, IEEE Trans. Neural Syst. Rehabil. Eng. 14, pp. 174–179, 2006.4. C. Menon, C. de Negueruela, J. del R. Milla´n, O. Tonet, F. Carpi, M. Broschart,P. Ferrez, A. Buttfield, F. Tecchio, F. Sepulveda, L. Citi, C. Laschi, M. Tombini,P. Dario, P. M. Rossini, and D. de Rossi, Prospects of brain-machine in-terfaces for space system control, Acta Astronaut. 64, pp. 448–456, 2009.doi:10.1016/j.actaastro.2008.09.0085. F. Gala´n, M. Nuttin, E. Lew, P. W. Ferrez, G. Vanacker, J. Philips, andJ. del R. Milla´n, A brain-actuated wheelchair: asynchronous and non-invasive brain-computer interfaces for continuous control of robots, Clin. Neurophysiol. 119,pp. 2159–2169, 2008. doi:10.1016/j.clinph.2008.06.0016. J. del R. Milla´n, F. Renkens, J. Mourin˜o, and W. Gerstner, Noninvasive brain-actuated control of a mobile robot by human EEG, IEEE Trans. Biomed. Eng. 51,pp. 1026–1033, 2004. doi:10.1109/TBME.2004.8270867. P. W. Ferrez and J. del R. Milla´n, Error-related EEG potentials generated duringsimulated brain-computer interaction, IEEE Trans. Biomed. Eng. 55, pp. 923–929,2008. doi:10.1109/TBME.2007.9080838. R. Chavarriaga, P. W. Ferrez, and J. del R. Milla´n, To err is human: learning fromerror potentials in brain-computer interfaces, Proc. Int’l Conf. Cogn. Neurodyn.,pp. 777–782, 2007.9. X. Perrin, R. Chavarriaga, F. Colas, R. Siegwart, and J. del R. Milla´n, Brain-coupled interaction for semi-autonomous navigation of an assistive robot, Robot. Au-ton. Syst. In press.10. K. Fo¨rster, A. Biasiucci, R. Chavarriaga, J. del R. Milla´n, D. Roggen, andG. Tro¨ster, On the use of brain decoded signals for online user adaptive gesture recog-nition systems, 8th Int’l Conf. Pervas. Comput. (Pervasive 2010), 2010. Avail-able from http://infoscience.epfl.ch/record/143441.c© 2010 PerAda

Context-Aware Brain-Computer Interfaces

http://infoscience.epfl.ch/record/150410/files/ChavarriagaMi10a.pdf

Context-Aware Brain-Computer Interfaces

Abstract

Similar works

Full text

Available Versions

Infoscience - École polytechnique fédérale de Lausanne