Desain, Peter

Farquhar, Jason

Haselager, Pim

Hesse, Christian

Schaefer, Rebecca

English

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      What BCI research needsCitation for published version:Desain, P, Farquhar, J, Haselager, P, Hesse, C & Schaefer, R 2008, 'What BCI research needs'. inProceedings of ACM CHI 2008 Conference on Human Factors in Computing Systems. CHI 2008, Florence,Italy, 5-10 May.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Preprint (usually an early version)Published In:Proceedings of ACM CHI 2008 Conference on Human Factors in Computing SystemsPublisher Rights Statement:© Desain, P., Farquhar, J., Haselager, P., Hesse, C., & Schaefer, R. (2008). What BCI research needs. InProceedings of ACM CHI 2008 Conference on Human Factors in Computing Systems.General rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015What BCI research needs  Peter Desain1, Jason Farquhar3, Pim Haselager1, Christian Hesse2, Rebecca Schaefer1 1 Cognitive Artificial Intelligence, NICI, Radboud University, Nijmegen, The Netherlands 2 F.C. Donders Centre for Cognitive Neuroimaging, Radboud University, Nijmegen, The Netherlands 3 Max Planck Institute for Biological Cybernetics, Tübingen, Germany  Introduction The state of the art of Brain Computer Interfacing, though promising, is still far from what one would need for fast and reliable control of games and interfaces. A mature non-invasive systems is e.g. the discrete P300 visual speller, in which alternating rows and columns of a letter matrix flash and gaze and attention is directed by the user to a specific letter (Sellers & Donchin, 2006). The EEG signal is analysed, and in a handful of flashes the selection can be determined with high accuracy. But even in optimizations of the flash regime (Hill, Farquhar, Martens, Biessmann & Schölkopf, 2007) the maximum data rate is still very low. And this BCI may not be independent, i.e. it needs gaze direction and cannot be controlled by mental activity alone. A second example of a mature BCI, for the continuous case, is the neuro-feedback trained slow cortical potential or e.g. theta-power for cursor control. (Birbaumer, 1977, Birbaumer et al., 1999). This BCI is also only able to achieve a very modest data rate of control in one dimension. Why is it that, in all these years of development, not more progress has been achieved? What makes the problem so complex and the signals that hard to decipher? Should we continue to build upon these examples and try to improve them? Or would we need new approaches? What are the fundamental issues that need addressing? Below we identify a series of topics that need consideration and may lead to fundamentally different, and better, brain computer interfaces.  Other mental tasks First of all, the source data have a big impact on the ease of analysis. Next to the currently developed systems that use mainly motor imagery, other mental tasks have only begun to emerge in a limited number of systems. Imagining different things, such as navigating through a known place, and musical imagery have been reported on, as well as selective attention (ie. Curran et al., 2004, Müller-Putz et al., 2006), but many others may be thought of. For instance, imagined or internal speech may tap into linguistic areas of the brain, visual and tactile imagery may evoke specific responses, and pain imagery might also be an intuitive task. Other types of cognitive processing such as spatial attention may be utilized by including imagined space, selective attention to body parts or imagined posture into the task.  Areas of research that are starting to emerge make use of selective attention between parallel streams, as this is specifically a mental function that we can perform particularly well. In this paradigm, different modalities may be used, such as auditory, tactile and visual. Multimodal stimuli may perceptually strengthen the stimulus and thus make it easier to attend to. As time is the most precise dimension in measuring brain activity with EEG, another improvement could be made by temporally structuring the different tasks. In this way, the analysis methods can also focus on specific time points and time intervals. E.g. selective attention may also be used serially, making use of fluctuations of attention over time. By using timing and rhythm, cognitive processes are also tapped into that are very well developed in human cognition. The use of imagined movement in most current BCI systems is not time locked, and quite unspecified. Making this movement rhythmic offers an improvement of the signal. Other large areas of cognitive functioning remain untapped by BCI research. Memory access is such a fundamental process that traces of neural activity thereof must be present in the signal. For aphasic patients, the BCI that recognizes failed lexical access (the tip-of-the-tongue phenomenon) and provides the proper word is e.g. not yet realistic, given our current state of understanding, but it is stimulating to try to understand what needs to be known before we can design such a device. Moving closer to detecting thoughts, next to internal (imagined) speech we can even think of imagined abstract semantic or affective items, like a affirmative/negative response without activating the actual words yes/no. Numerosity is another candidate, e.g. the abstract idea of duple or triple-ness without the word two or three. For all of these tasks the representation in brain signals is not yet understood well enough.  Better understanding and representation of signal features In accordance with the biophysics of EEG signal generation, the measured multi-channel signals can be assumed to be a linear mixture of several underlying source signals, of which some reflect brain activity of interest, e.g., task-related amplitude modulation of ongoing neuronal oscillations or evoked activity. Hence, the central signal processing challenge for any BCI approach is to identify, extract and encode the activity patterns of task-relevant neuronal sources, and to map these onto the output (command) channels of the BCI system (i.e., signal classification). The mixing of various brain signals and artifacts at the sensors gives rise to strong spatial correlations in the recorded EEG signals, which limit the amount of unique information that can be extracted from individual sensors, even when the target signals have reasonably well defined frequency characteristics. Spatial filtering techniques provide one possible means of increasing the amount of information that can be gleaned from multi-channel EEG, specifically by transforming the signals in such a way that spatial correlations at the output are removed (e.g., as in independent component analysis) or minimized (e.g., as in adaptive beam-forming approaches based on biophysical source models or scalp topographies estimated in a supervised way, e.g., using common spatial patterns. The temporal dynamics of the spatially filtered signals (i.e., extracted sources) can then be further characterized by means of conventional time-frequency decomposition methods, e.g., based on the spectrogram or discrete wavelet transforms. In this context it could also be of interest to consider the temporal relationships (e.g., delays or phase shifts) between the extracted sources (possibly with an external pacing signal), and how these vary as a function of mental state. While combined spatial filtering and time-frequency decomposition provides a rich characterization of the EEG signal and a large source of features for classification, it raises the issue of how to pick the relevant subset of features, i.e. those that drive the classification. Selection of “good” spatial and time-frequency features is important not only for maximizing classification performance, but also facilitates computationally efficient and fast extraction of those features directly from the multi-channel EEG without the need for separate pre-processing steps. In turn, fast feature extraction and classification helps to increase the refresh rate of the BCI system and the number of classes that can be analyzed online, and this ultimately helps to boost bit-rates. Statistical Machine Learning provides many techniques for reliability performing this feature selection in for BCI (Lal et al. 2004; Farquhar et al. 2006).  The development of such techniques, particularly when combined with feedback for cooperative user feature selection and training, offers the potential to adapt the BCI to each users particularly abilities. Next to understanding the representation of natural tasks, it is also possible to exploit the manipulation of stimuli further, watermarking or tagging them with recognizable patterns such that attention and recognition can be tracked. Frequency tagging is only the start of this line of research. Developing watermarks that are less easy overshadowed by the naturally occurring oscillations in the brain would be a next step.  Feedback and mutual adaptation In many ways the human brain is the most flexible and powerful learning machine currently in existence. How a child learns to control its own muscles with only slight feedback clearly demonstrates this power. In order for BCI systems to utilize this ability to improve their performance we require: firstly, training paradigms and feedback systems to show the user their current brain states and to learn how to control them; and secondly, cooperative learning systems to track and adapt to the changing user state. One way to address this within BCI would be start training on the basis of a mental pendant of motor babbling observed in young infants (e.g. the seemingly random moving around of arms and feet). Instead of starting out to perform a well-defined and highly regimented task, the user would be invited, at an early stage, to a (controlled) form of ‘simply trying’ while interacting with the interface. This way, users could discover and evaluate the consequences of their mental actions in a more natural and self-initiated way. It is currently unclear how to present feedback effectively to allow the user to search for their most effective control dimension. Even current "simple" imagined movement and ERP based BCIs have a vast number of dimensions which the user could control, e.g. spatial location, temporal trace, spectral power. Presenting all these possibilities to the user on-line, for example via a spectrogram/waterfall display, is unlikely to work due to cognitive overload. Thus some sort of mutual adaptation is necessary, where the system selects for display only those dimensions the user can best control. Such a system should also remain adaptive to the users changing abilities, to both allow finer control in use and to present them with more complex feedback as their abilities improve. One issue yet to be addressed with such a system however is how to ensure both learning systems operate in a cooperative fashion. In addition to deciding 'what' feedback should be presented, 'how' it is presented is clearly important.  Ideally, the feedback should be "natural" such that correspondence between the mental task and feedback is obvious, for example using right hand tactile stimulation in response to right hand motor imagery.  High dimensional output  At present continuous BCI's, even those in which the user receives lots of training in a neuro-feedback situation, only support up to two or three dimensions of independent control (Schalk et al. 2007). It is rather cynical that one of the most high-dimensional systems we know cannot be made to control, say a robot arm, with a relatively minor number of degrees of freedom. We think that subsequent searching for controllable dimension in the brain's signal space and shaping and training the control of them one by one in a mutually adaptive BCI, might give a solution. Independent decoding of dimensions allows for very complex control without the need for collecting complex training data of the whole space. Decomposition For discrete BCI's that output symbolic classes a similar concern exists. The power would be greatly increased if recursive ways of composing recognized smaller units into larger wholes is found. For certain domains, like language, in which phonemes compose into syllables, syllables into words etc., a structural isomorhism is not very likely, as at every level extra information (syntax, semantics) becomes available: the whole is more than the sum of the parts. However, because it is likely that different brain areas are involved in computation at the different levels of representation, it may still be possible to access and decode these levels separately. The advantage is that we can move from recognition at the high level, in which training data for all classes has to be available, to recognition at a lower level, with possibly fewer symbols to decode. For certain tasks, like independent finger movements, a structural isomorphism may model the data already to a larger extend, and decoding combinations of fingers may be factored into decoding the individual movements.  Conclusion Though large steps towards reliable BCI's have been made, much progress is still needed if we want to apply BCI in everyday use in consumer products. Luckily there are many areas and aspects that are promising and are still hardly explored.    References Birbaumer, N. (1977). Operant enhancement of EEG-theta activity: aspiration and reality. In J.Beatty & H. Legewie (Eds.), Biofeedback and Behavior (pp. 135-146). New York: Plenum Press. Birbaumer, N., N. Ghanayim, T. Hinterberger, I. Iversen, B. Kotchoubey, A. Kübler, J.  Perelmouter, E. Taub, E. & H. Flor, (1999). A spelling device for the paralysed. Nature, 398, 297-298 Curran, E., P. Sykacek, M. Stokes, S.J. Roberts, W. Penny, I. Johnsrude & O. Owen, (2004). Cognitive tasks for driving a brain-computer interfacing system: A pilot study. Ieee Transactions on Neural Systems and Rehabilitation Engineering, 12(1), 48-54. Farquhar, J., J. Hill & B. Schölkopf (2006). Learning optimal EEG features across time, frequency and space. Twentieth Annual Conference on Neural Information Processing Systems : NIPS 2006, Vancouver, Canada. Lal, T. N., M. Schröder, T. Hinterberger, J. Weston, M. Bogdan, N. Birbaumer & B. Schölkopf (2004). Support Vector Channel Selection in BCI. IEEE Transactions on Biomedical Engineering 51(6), 1003-1010 Müller-Putz, G.R., Scherer, R., Neuper, C. & Pfurtscheller, G. (2006). Steady-state somatosensory evoked potentials: suitable brain signals for brain-computer interfaces? IEEE Transactions on Neural Systems and Rehabilitation Engineering, 14(1), 30-37 Schalk, G., J. Kubanek, K.J. Miller, N.R. Anderson, E.C. Leuthardt, J.G. Ojemann, D. Limbrick, D. Moran, A. Gerhard, J.R. Wolpaw (2007). Decoding two dimensional movement trajectories using electrocorticographic signals in humans, Journal of Neural Engineering, 4, pp 264-275. Sellers, E.W., & E. Donchin (2006). A p300-based brain computer-interface: Initial tests by ALS patients. Clin Neurophysiol. 117(3): 538-548. 

A p300-based brain computer-interface: Initial tests by ALS patients.

A spelling device for the paralysed.

Decoding two dimensional movement trajectories using electrocorticographic signals in humans,

Learning optimal EEG features across time, frequency and space.

Operant enhancement of EEG-theta activity: aspiration and reality.

Support Vector Channel Selection in BCI.

What BCI research needs

What BCI research needs

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer