The central question of this Frontiers Research Topic is: What can we learn from brain and other physiological signals about an individual's cognitive and affective state and how can we use this information? This question reflects three important issues which are addressed by the 22 articles in this volume: (1) the combination of central and peripheral neurophysiological measures; (2) the diversity of cognitive and affective processes reflected by these measures; and (3) how to apply these measures in real world applications

Brouwer, Anne-Marie

Erp, Jan van

Zander, Thorsten O.

PubMed

The central question of this Frontiers Research Topic is: What can we learn from brain and other physiological signals about an individual's cognitive and affective state and how can we use this information? This question reflects three important issues which are addressed by the 22 articles in this volume: (1) the combination of central and peripheral neurophysiological measures; (2) the diversity of cognitive and affective processes reflected by these measures; and (3) how to apply these measures in real world application

Erp, J.B.F. van

Brouwer, A.M.

Zander, T.O.

NARCIS 

Introduction to using neurophysiological signals that reflect cognitive or affective state (Editorial)

Jan B. F. van Erp

Anne-Marie Brouwer

Thorsten O. Zander

Frontiers - Publisher Connector

Editorial: Using neurophysiological signals that reflect cognitive or affective state

van Erp, Johannes Bernardus Fransiscus

University of Twente Research Information

EDITORIALpublished: 29 May 2015doi: 10.3389/fnins.2015.00193Frontiers in Neuroscience | www.frontiersin.org 1 May 2015 | Volume 9 | Article 193Edited and reviewed by:Cuntai Guan,Institute for Infocomm Research,Singapore*Correspondence:Jan B. F. van Erp,jan.vanerp@tno.nlSpecialty section:This article was submitted toNeuroprosthetics,a section of the journalFrontiers in NeuroscienceReceived: 08 April 2015Accepted: 16 May 2015Published: 29 May 2015Citation:van Erp JBF, Brouwer A-M andZander TO (2015) Editorial: Usingneurophysiological signals that reflectcognitive or affective state.Front. Neurosci. 9:193.doi: 10.3389/fnins.2015.00193Editorial: Using neurophysiologicalsignals that reflect cognitive oraffective stateJan B. F. van Erp 1, 2*, Anne-Marie Brouwer 1 and Thorsten O. Zander 31 TNO Human Factors, Soesterberg, Netherlands, 2Human Media Interaction, University of Twente, Enschede, Netherlands,3 Team PhyPA, Technical University Berlin, Berlin, GermanyKeywords: neurophysiology, cognitive state, mental state, affective state, EEG, Brain-Computer Interface,psychophysiology, physiological computingIntroductionThe central question of this Frontiers Research Topic is: What can we learn from brain and otherphysiological signals about an individual’s cognitive and affective state and how can we use thisinformation? This question reflects three important issues which are addressed by the 22 articlesin this volume: (1) the combination of central and peripheral neurophysiological measures; (2) thediversity of cognitive and affective processes reflected by these measures; and (3) how to apply thesemeasures in real world applications.Neurophysiological Measures and Real World ApplicationsLet us first look at the last issue as it is an important driver of the current research and dictateschoices related to the first two, for instance when a specific application requires sensor technologyto be portable and easy to set up with little calibration time (Gerjets et al., 2014; Huang et al.,2014; Estepp and Christensen, 2015). The authors of this Research Topic describe a wide variety ofapplications. Many studies follow a so-called passive Brain-Computer Interface approach (ZanderandKothe, 2011): assessing typically covert aspects of the user state without interfering with the taskthe user is doing. Applications addressed in this volume include improved learning and trainingenvironments (Gerjets et al., 2014; Stikic et al., 2014), adaptive vehicle interfaces (Dijksterhuis et al.,2013; Touryan et al., 2014; Wang et al., 2014), classifying concealed information (Farwell et al.,2014), identifying sleep stages (Huang et al., 2014), and non-verbal communication for patients(Kashihara, 2014). Making these applications viable is not trivial. It requires experiments outsidea well-controlled lab environment as initiated in several studies here (Dijksterhuis et al., 2013;Betella et al., 2014; Huang et al., 2014; Lin et al., 2014; Stuiver and Mulder, 2014) and knowledgeand technological advancements in the area of classification algorithms that require little trainingdata, are robust against environmental and physiological artifacts in the recorded signal, are ableto work in real-time and on single trial data, and generalize over tasks. Unlike a lab experimentwhere participants are usually instructed to sit still and to perform one single task, users in realworld situations will usually be in motion, subject to changing cognitive load or emotional state,and/or be multitasking (Van Erp et al., 2012). Several authors focus on developing methods ableto cope with these added complexities in life-like situations. For example, Betella et al. (2014)assesses the reliability of psychophysiological signals reflecting emotional state when people arewalking and gesticulating. Mühl et al. (2014) look into the effect of emotional state on workloadmeasures, and show that cross-context classifier training is able to compensate for accuracydecline caused by changes in emotional state (especially so for indices in the frequency domain).van Erp et al. Neurophysiological signals reflecting mental stateTouryan et al. (2014) are able to model the time-on-task decrements in performance by analyzing neural activity.Chavarriaga et al. (2014), Vijgh et al. (2014) and Wang et al.(2014) give examples of real time classification. Vijgh et al. (2014)aim to asses stress state and adjust stressor intensity in a gamingenvironment in real time, Wang et al. (2014) are able to detectthe effectiveness of a drowsiness warning system and continuoussignatures of fatigue, Chavarriaga et al. (2014) develop single-trialerror-related potential recognition in order to improve human-computer interaction, and Estepp andChristensen (2015) presentinitial proof that electrodes can be removed and replaced betweensessions without dramatic loss in performance. Gerjets et al.(2014), Stikic et al. (2014), Stuiver and Mulder (2014) andTouryan et al. (2014) tackle the question about generalizabilityover tasks. This is an important issue because if the modelsand algorithms are highly task-specific, it would mean that eachapplication has to start from scratch in choosing the optimalparameters, gathering data and building classification algorithms.Generic classifiers can possibly shorten the calibration time,increasing the usability and applicability. Stuiver and Mulder(2014) investigate and try to grasp differences in the physiologicalpatterns of an ambulance dispatcher task and driving. Stikic et al.(2014) and Touryan et al. (2014) focus on the validity of theiralgorithms for different tasks: marksmanship and golf (Stikicet al., 2014), and driving and perceptual discrimination (Touryanet al., 2014). It is not surprising that many authors look at ways tofurther improve signal processing and classification techniquesincluding neural networks (Casson, 2014; Stikic et al., 2014),feature fusion (Putze et al., 2014), and elastic nets (Hogervorstet al., 2014).Constructs of InterestThe wide range of foreseen applications also illustrate thebroadness of the constructs of interest. Workload, stress, andemotion are classic examples, but there is also an increasinginterest in the ability to determine what information thebrain is processing (Pineda et al., 2013; Lin et al., 2014)and for instance whether this is visual or auditory (Putzeet al., 2014), which in turn links to the multitasking challengeof real world applications. Additional contributions to thisarea come from studies into new paradigms to evoke specificstates as discussed by Gerjets et al. (2014). For instance,Vijgh et al. (2014) describe an automated stress inductionand control application and Brouwer and Hogervorst (2014)describe an elegant, simple and ethically acceptable way ofinducing mental stress that results in physiological responsescomparable to those induced by the Trier Social StressTest.Combining Central and PeripheralNeurophysiological MeasuresThe complexity of the constructs to be measured, and the factthat (neuro-) physiological indices only provide indirect accessto them forces this field to explore and combine measures fromdifferent sources. Quite novel are particular combinations ofcentral and peripheral measures and combinations of differentcentral sensors (more specifically EEG and NIRS). Strait andScheutz (2014) start with an interesting analysis of what we canand cannot (yet) do with fNIRS and Putze et al. (2014) providean example that fusing EEG and fNIRS features can increase theaccuracy of classification. Stikic et al. (2014) combine ECG andEEG derived measures to correlate physiological changes withtraining progress, and Hogervorst et al. (2014) determine theworkload classification accuracy of EEG, ECG, skin conductanceand eye-based measures, alone and in combination. This studycontributes to the long-lasting dispute about the best workloadmeasures in favor of EEG-derived indices. More particular,information derived from a single electrode location (Pz) turnsout to be an adequate workload predictor. Finally, papers alsodemonstrate the use of a wide variety of indices from a singlesource, for instance EEG is analyzed both in the time domain(P3 event related potentials, Farwell et al., 2014 and error-relatedpotentials, Chavarriaga et al., 2014) and the frequency domain (asworkload index, Mühl et al., 2014 or in both, Hogervorst et al.,2014). Finally, Gerjets et al. (2014) and Zander and Jatzev (2012)argue that contextual information can be a useful addition toneurophysiological measures.Concluding RemarksThe papers in this special issue show that the field of applied(neuro-) physiology is progressing in many aspects, especiallyin EEG-based passive Brain-Computer Interfaces. This is not tosay that there are no remaining issues, but the field seems to bewell-aware of the challenges it is facing (Gerjets et al., 2014; Straitand Scheutz, 2014; Brouwer et al., 2015) and the knowledge andtechnological breakthroughs it requires on the way to real-worldapplications. Also in order here is a word of caution: there aremany potential pitfalls in using neurophysiological measures aslisted by (Gerjets et al., 2014; Brouwer et al., 2015). However,and as also indicated, there are ways around them. The stepscurrently taken in applied neurophysiology also touch on otherissues, such as ethics, acceptance of the technology by the generalpublic, privacy of users, and the possible effects that these kindsof applications may have on society as a whole. These are notcovered in this issue, but should also be borne in mind (Van Erpet al., 2012).ReferencesBetella, A., Zucca, R., Cetnarski, R., Greco, A., Lanatà, A., Mazzei, D., et al. (2014).Inference of human affective states from psychophysiological measurementsextracted under ecologically valid conditions. Front. Neurosci. 8:286. doi:10.3389/fnins.2014.00286Brouwer, A.-M., and Hogervorst, M. (2014). A new paradigm to inducemental stress: the sing-a-song stress test (SSST). Front. Neurosci. 8:224. doi:10.3389/fnins.2014.00224Brouwer, A.-M., Zander, T. O., Van Erp, J. B. F., Korteling, J. E., andBronkhorst, A. W. (2015). Using neurophysiological signals thatreflect cognitive or affective state: six recommendations to avoidFrontiers in Neuroscience | www.frontiersin.org 2 May 2015 | Volume 9 | Article 193van Erp et al. Neurophysiological signals reflecting mental statecommon pitfalls. Front. Neurosci. 9:136. doi: 10.3389/fnins.2015.00136Casson, A. J. (2014). Artificial Neural Network classification of operator workloadwith an assessment of time variation and noise-enhancement to increaseperformance. Front. Neurosci. 8:372. doi: 10.3389/fnins.2014.00372Chavarriaga, R., Sobolewski, A., and Millán, Jdel. R. (2014). Errare machinale est:the use of error-related potentials in brain-machine interfaces. Front. Neurosci.8:208. doi: 10.3389/fnins.2014.00208Dijksterhuis, C., De Waard, D., Brookhuis, K., Mulder, B., and De Jong, R. (2013).Classifying visuomotor workload in a driving simulator using subject specificspatial brain patterns. Front. Neurosci. 7:149. doi: 10.3389/fnins.2013.00149Estepp, J. R., and Christensen, J. C. (2015). Electrode replacement doesnot affect classification accuracy in dual-session use of a passive brain-computer interface for assessing cognitive workload. Front. Neurosci. 9:54. doi:10.3389/fnins.2015.00054Farwell, L. A., Richardson, D. C., Richardson, G. M., and Furedy, J. J. (2014).Brain fingerprinting classification concealed information test detects USNavy military medical information with P300. Front. Neurosci. 8:410. doi:10.3389/fnins.2014.00410Gerjets, P., Walter, C., Rosenstiel, W., Bogdan, M., and Zander, T. O. (2014).Cognitive state monitoring and the design of adaptive instruction in digitalenvironments: lessons learned from cognitive workload assessment usinga passive brain-computer interface approach. Front. Neurosci. 8:385. doi:10.3389/fnins.2014.00385Hogervorst, M. A., Brouwer, A.-M., and Van Erp, J. B. F. (2014). Combiningand comparing EEG, peripheral physiology and eye-related measuresfor the assessment of mental workload. Front. Neurosci. 8:322. doi:10.3389/fnins.2014.00322Huang, C.-S., Lin, C.-L., Ko, L.-W., Liu, S.-Y., Su, T.-P., and Lin, C.-T.(2014). Knowledge-based identification of sleep stages based on twoforehead electroencephalogram channels. Front. Neurosci. 8:263. doi:10.3389/fnins.2014.00263Kashihara, K. (2014). A brain–computer interface for potential nonverbal facialcommunication based on eeg signals related to specific emotions. Front.Neurosci. 8:244. doi: 10.3389/fnins.2014.00244Lin, Y.-P., Yang, Y.-H., and Jung, T.-P. (2014). Fusion of Electroencephalogramdynamics and musical contents for estimating emotional responses in musiclistening. Front. Neurosci. 8:94. doi: 10.3389/fnins.2014.00094Mühl, C., Jeunet, C., and Lotte, F. (2014). EEG-based workload estimation acrossaffective contexts. Front. Neurosci. 8:114. doi: 10.3389/fnins.2014.00114Pineda, J. A., Grichanik, M., Williams, V., Trieu, M., Chang, H., and Keysers, C.(2013). EEG sensorimotor correlates of translating sounds into actions. Front.Neurosci. 7:203. doi: 10.3389/fnins.2013.00203Putze, F., Hesslinger, S., Tse, C.-Y., Huang, Y. Y., Herff, C., Guan, C., et al. (2014).Hybrid fNIRS-EEG based classification of auditory and visual perceptionprocesses. Front. Neurosci. 8:373. doi: 10.3389/fnins.2014.00373Stikic, M., Berka, C., Levendowski, D., Rubio, R. F., Tan, V., Korszen, S., et al.(2014). Modeling temporal sequences of cognitive state changes based ona combination of EEG-engagement, EEG-workload, and heart rate metrics.Front. Neurosci. 8:342. doi: 10.3389/fnins.2014.00342Strait, M., and Scheutz, M. (2014). What we can and cannot (yet) dowith functional near infrared spectroscopy. Front. Neurosci. 8:117. doi:10.3389/fnins.2014.00117Stuiver, A., and Mulder, B. (2014). Cardiovascular state changes in simulated workenvironments. Front. Neurosci. 8:399. doi: 10.3389/fnins.2014.00399Touryan, J., Apker, G., Lance, B. J., Kerick, S. E., Ries, A. J., and McDowell, K.(2014). Estimating endogenous changes in task performance from EEG. Front.Neurosci. 8:155. doi: 10.3389/fnins.2014.00155Van Erp, J. B. F., Lotte, F., and Tangermann, M. (2012). Brain-computerinterfaces: beyond medical applications. IEEE Comput. 45, 26–34. doi:10.1109/MC.2012.107Vijgh, B., Van Der Beun, R.-Jan., Van Rood, M., and Werkhoven, P. (2014).GASICA: generic automated stress induction and control application designof an application for controlling the stress state. Front. Neurosci. 8:400. doi:10.3389/fnins.2014.00400Wang, Y.-T., Huang, K.-C., Wei, C.-S., Huang, T.-Y., Ko, L.-W., Lin, C.-T., et al.(2014). Developing an EEG based on-line closed-loop lapse detection andmitigation system. Front. Neurosci. 8:321. doi: 10.3389/fnins.2014.00321Zander, T. O., and Jatzev, S. (2012). Context-aware brain–computer interfaces:exploring the information space of user, technical system and environment.J. Neural Eng. 9:016003. doi: 10.1088/1741-2560/9/1/016003Zander, T. O., and Kothe, C. (2011). Towards passive brain–computerinterfaces: applying brain–computer interface technology to human–machinesystems in general. J. Neural Eng. 8:025005. doi: 10.1088/1741-2560/8/2/025005Conflict of Interest Statement: The authors declare that the research wasconducted in the absence of any commercial or financial relationships that couldbe construed as a potential conflict of interest.Copyright © 2015 van Erp, Brouwer and Zander. This is an open-access articledistributed under the terms of the Creative Commons Attribution License (CC BY).The use, distribution or reproduction in other forums is permitted, provided theoriginal author(s) or licensor are credited and that the original publication in thisjournal is cited, in accordance with accepted academic practice. No use, distributionor reproduction is permitted which does not comply with these terms.Frontiers in Neuroscience | www.frontiersin.org 3 May 2015 | Volume 9 | Article 193

English

Editorial: Using neurophysiological signals that reflect cognitive or affective state.

Editorial: Using neurophysiological signals that reflect cognitive or affective state

Abstract

Similar works

Full text

Available Versions

NARCIS

Frontiers - Publisher Connector

University of Twente Research Information

NARCIS