Pervasive brain monitoring and data sharing based on multi-tier distributed computing and linked data technology

EEG-based Brain-computer interfaces (BCI) are facing grant challenges in their real-world applications. The technical difficulties in developing truly wearable multi-modal BCI systems that are capable of making reliable real-time prediction of users’ cognitive states under dynamic real-life situations may appear at times almost insurmountable. Fortunately, recent advances in miniature sensors, wireless communication and distributed computing technologies offered promising ways to bridge these chasms. In this paper, we report our attempt to develop a pervasive on-line BCI system by employing state-of-art technologies such as multi-tier fog and cloud computing, semantic Linked Data search and adaptive prediction/classification models. To verify our approach, we implement a pilot system using wireless dry-electrode EEG headsets and MEMS motion sensors as the front-end devices, Android mobile phones as the personal user interfaces, compact personal computers as the near-end fog servers and the computer clusters hosted by the Taiwan National Center for High-performance Computing (NCHC) as the far-end cloud servers. We succeeded in conducting synchronous multi-modal global data streaming in March and then running a multi-player on-line BCI game in September, 2013. We are currently working with the ARL Translational Neuroscience Branch and the UCSD Movement Disorder Center to use our system in real-life personal stress and in-home Parkinson’s disease patient monitoring experiments. We shall proceed to develop a necessary BCI ontology and add automatic semantic annotation and progressive model refinement capability to our system

Hierarchical Event Descriptors (HED): Semi-Structured Tagging for Real-World Events in Large-Scale EEG.

Real-world brain imaging by EEG requires accurate annotation of complex subject-environment interactions in event-rich tasks and paradigms. This paper describes the evolution of the Hierarchical Event Descriptor (HED) system for systematically describing both laboratory and real-world events. HED version 2, first described here, provides the semantic capability of describing a variety of subject and environmental states. HED descriptions can include stimulus presentation events on screen or in virtual worlds, experimental or spontaneous events occurring in the real world environment, and events experienced via one or multiple sensory modalities. Furthermore, HED 2 can distinguish between the mere presence of an object and its actual (or putative) perception by a subject. Although the HED framework has implicit ontological and linked data representations, the user-interface for HED annotation is more intuitive than traditional ontological annotation. We believe that hiding the formal representations allows for a more user-friendly interface, making consistent, detailed tagging of experimental, and real-world events possible for research users. HED is extensible while retaining the advantages of having an enforced common core vocabulary. We have developed a collection of tools to support HED tag assignment and validation; these are available at hedtags.org. A plug-in for EEGLAB (sccn.ucsd.edu/eeglab), CTAGGER, is also available to speed the process of tagging existing studies

Preparing Laboratory and Real-World EEG Data for Large-Scale Analysis: A Containerized Approach.

Large-scale analysis of EEG and other physiological measures promises new insights into brain processes and more accurate and robust brain-computer interface models. However, the absence of standardized vocabularies for annotating events in a machine understandable manner, the welter of collection-specific data organizations, the difficulty in moving data across processing platforms, and the unavailability of agreed-upon standards for preprocessing have prevented large-scale analyses of EEG. Here we describe a "containerized" approach and freely available tools we have developed to facilitate the process of annotating, packaging, and preprocessing EEG data collections to enable data sharing, archiving, large-scale machine learning/data mining and (meta-)analysis. The EEG Study Schema (ESS) comprises three data "Levels," each with its own XML-document schema and file/folder convention, plus a standardized (PREP) pipeline to move raw (Data Level 1) data to a basic preprocessed state (Data Level 2) suitable for application of a large class of EEG analysis methods. Researchers can ship a study as a single unit and operate on its data using a standardized interface. ESS does not require a central database and provides all the metadata data necessary to execute a wide variety of EEG processing pipelines. The primary focus of ESS is automated in-depth analysis and meta-analysis EEG studies. However, ESS can also encapsulate meta-information for the other modalities such as eye tracking, that are increasingly used in both laboratory and real-world neuroimaging. ESS schema and tools are freely available at www.eegstudy.org and a central catalog of over 850 GB of existing data in ESS format is available at studycatalog.org. These tools and resources are part of a larger effort to enable data sharing at sufficient scale for researchers to engage in truly large-scale EEG analysis and data mining (BigEEG.org)

Towards continuous and real-time attention monitoring at work: reaction time versus brain response

Continuous and objective measurement of the user attention state still represents a major challenge in the ergonomics research. Recently available wearable electroencephalography (EEG) opens new opportunities for objective and continuous evaluation of operators' attention, which may provide a new paradigm in ergonomics. In this study, wearable EEG was recorded during simulated assembly operation, with the aim to analyse P300 event-related potential component, which provides reliable information on attention processing. In parallel, reaction times (RTs) were recorded and the correlation between these two attention-related modalities was investigated. Negative correlation between P300 amplitudes and RTs has been observed on the group level (p lt .001). However, on the individual level, the obtained correlations were not consistent. As a result, we propose the P300 amplitude for accurate attention monitoring in ergonomics research. On the other hand, no significant correlation between RTs and P300 latency was found on group, neither on individual level. Practitioner Summary: Ergonomic studies of assembly operations mainly investigated physical aspects, while mental states of the assemblers were not sufficiently addressed. Presented study aims at attention tracking, using realistic workplace replica. It is shown that drops in attention could be successfully traced only by direct brainwave observation, using wireless electroencephalographic measurements.This is the peer-reviewed version of the article: Mijović, P.; Ković, V.; De Vos, M.; Macuzić, I.; Todorović, P.; Jeremić, B.; Gligorijević, I. Towards Continuous and Real-Time Attention Monitoring at Work: Reaction Time versus Brain Response. Ergonomics 2017, 60 (2), 241–254. [https://doi.org/10.1080/00140139.2016.1142121

Development and implementation of multimodal system for attention monitoring in naturalistic work environments.

Kako tehnologija stalno napreduje, industrijske nesreće koje se vezuju za neispravnost tehničkih sistema su skoro skroz umanjene. Iz tog razloga, ljudska greška se smatra uzročnikom oko 80% nesreća u industriji. Jedan od glavnih uzročnika ljudske greške je limitirana mentalna izdržljivost ljudskih operatera, koja uzrokuje pad u pažnji radnika i posledično vodi do grešaka u radu. Klasične ergonomske metode koje se koriste za procenu kognitivnog stanja čoveka su uglanom kvalitativne i subjektivne, i prema tome su prilično nepouzdane. Iz tog razloga, psihofiziološki senzori su počeli da se primenjuju u ergonomiskim istraživanjima, sa ciljm da obezbede objektivne i kvantitativne mere radnikovog kognitivnog stanja. Prateći taj trend, neuroergonomija se pojavila kao naučna pod-disciplina ergonomije. Prednost korišćenja neuroergonomskih metoda, je u tome što neuroergonomija istražuje funkcionalnu zavisnost između dinamike mozga i bihevioralnih parametara i tako zaobilazi teoretske veze koje opisuju korelaciju između ovih parametara, a koje su korišćene u ergonomiji. Ova disertacija predstavlja naučni okvir za multimodalni sistem koji je predložen da se koristi za praće pažnje radnika i koji koristi psihofiziološke senzore i bihevijoralna merenja. Sistem se sastoji od psihofizioloških senzora, kao što su: galvanski reakciju kože (eng. galvanic skin response - GSR), merenje otkucaja srca (eng. heart rate –HR) i elekroencefalografiju (eng. Electroencephalography – EEG); bihevioralne modalitete kao što su: Vremena reakcija (eng. reaction times – RTs) i senzore za praćenje pokreta (eng. motion capture – MoCap), “Kinect” the “Leap Motion”. Iako je predstavljen okvir za snimanje pomenutih modaliteta u realnom vremenu, ova disertacija je fokusirana na rezultate koji su dobijeni snimanjem EEG, RTs i Kinect modaliteta. Glavni cilj disertacije je istraživanje mogućnosti korišćenja savremenog prenosnog EEG-a u industrijskim uslovima, sa ciljem praćenja pažnje radnika. Prethodna istraživanja koja su koristila EEG su bila uglavnom obavljana u kontrolisanim laboratorijskim uslovima i zbog toga, nalazi iz tih studija se uzimaju sa određenom dozom rezerve. Da bi se snimio EEG u realnom radnom okruženju, radno mesto u kojem radnici sklapaju hidraulično crevo je verodostojno replicirano i subjekti u studiji su simulirali taj proces. Disertacija se sastoji od četiri eksperimentalne studije. U prvoj studiji, ispitivano je kako česte mikro-pauze utiču na nivo pažnje radnika, poredeći amplitude P300 Komponente evociranih kognitivnih potencijala (eng. event-related potential – ERP) pre i neposredno posle perioda mikro-pauze. Glavni nalaz je da mikro-pauze pozitivno utiču na nivo pažnje radnika i predloženo je njihovo uključenje u dnevne aktivnosti radnika. U drugoj studiji, istraživano je da li radnici imaju veći nivo pažnje ukoliko im je nametnuto sa kojom rukom da počnu sklapanje creva. Dve psihološke paradigme su bile predstavljene učesnicima u studiji, paralelno sa simuliranm akcijom sklapanja creva. U prvoj paradigmi, učesnici su mogli da izaberu da otpočnu operaciju sa bilo kojom rukom, dok su u drugoj bili uslovljeni da započnu operaciju rukom koja odgovara smeru strelice koja se prikazivala na ekranu ispred njih. Ovo istraživanje je otkrilo da su učesnici imali veći nivo pažnje u slučaju uslovljavanja kojom rukom da započnu operaciju, jer je amplituda P300 komponente bila značajno viša u poređenju sa slučajem kada su mogli slobodno da izaberu sa kojom rukom će započinjati zadatak.As technology is ever advancing, industrial accidents related to technological malfunctioning have been almost diminished, leaving the human error responsible for up to 80% of the remaining accidents. One of the main causes for this is limited mental endurance of human operators’, which causes the attention decline and consequently leads to an operating error. Classical ergonomics methods for assessing the operators’ cognitive state are still dependent on the subjective and qualitative methods, thus being unreliable. For that reason, in the recent years the psychophysiological sensors were included in the ergonomics research, with the aim of providing the objective and quantitative measures of the operators’ cognitive state. Following that path, the neuroergonomics emerged as a scientific discipline, which investigates the human brain functions in relation to performance at work. The advantage of using neuroergonomics is that it investigates the functional relationship between brain dynamics and behavioral parameters, thus avoiding theoretical constructs that describe the correlation between these two, and which are ubiquitously used in ergonomics research. The present dissertation introduces a framework for the multimodal attention monitoring system, utilizing psychophysiological and behavioral measurements. The multimodal system consists of psychophysiological sensors, such as galvanic skin response (GSR), heart rate (HR) sensor and electroencephalography (EEG), the behavioral modality of the reaction times (RTs), and the motion capture (MoCap) sensors Kinect and the Leap Motion. Although the framework for synchronous and real-time recording for all the sensors was provided, this thesis was focused solely on the results obtained from the EEG, RTs and Kinect recordings. The main aim of the presented dissertation is to investigate the possibility of utilization of the recently available wearable electroencephalography (EEG) in industrial setting, with the goal of the operator’s attention monitoring. Previously reported EEG studies that were concerned with the attention states of the operators were mainly confined to the strictly controlled laboratory conditions and therefore, the findings from these studies needed to be taken with the certain ambiguity. In order to record the EEG in naturalistic environment, specific workplace where operators’ assembly the hoses, used in hydraulic break systems in vehicles, was faithfully replicated and the participants in the studies simulated the manual assembly operations. The present dissertation consists of four experimental studies, where the first two were concerned with investigation how different work conditions influence the cognitive state of the operators’, i.e. the studies were concerned with the assembly task design. In the first study, the influence of the frequent micro-breaks on the cognitive state of the workers’ was investigated, by comparing the P300 event-related potential (ERP) amplitude prior and immediately following the micro-break period. It was found that the micro-breaks enhance the attention of the operators’ and the proposal for their inclusion in the regular work routine was made. Second study investigated the influence of hand alteration on the attention level of the operators’. For that aim, the participants in the study were presented with two distinct task: the one in which they could initiate the assembly operation with whichever hand they preferred, and the one in which they were conditioned with which hand they should initiate the operation. This study revealed that the instructed hand responding induces the higher attention, as assessed through the P300 component’s amplitude, compared to the experimental condition where the participants could freely choose the hand for the initiation of the assembly operation. Further, a framework for the on-line assessment of the operators’ cognitive state was provided. In the third experimental study, the propagation of the P300 component’s amplitude was observed and correlated with the RTs. On the group level, a negative correlation was found, confirming the previously reported finding. However, due to individual differences, the correlation on the individual level was inconsistent, emphasizing the necessity for the individualized EEG measurements for the reliable attention monitoring system. Finally, it was investigatetd whether the quantity of task unrelated movements corresponds to attention of the operator, as previously shown to be negatively related to the attention of operators’. For that aim, the concept of movement energy (ME) was introduced and correlated with EEG attention-related modalities. The initial finding from this study showed that the ME is negatively related to the EEG attention-related modalities and proved that the future attention monitoring system can be built based on these modalities

Neural correlates of prospective memory: an EEG and ICA approach

Have you ever entered a room and wondered ‘What am I supposed to do here?’ or have you ever forgotten to turn off the oven, hang your clothes to dry or make a phone call. These examples illustrate the relevance of ‘prospective memory’ or ‘delayed intentions’ in our daily life activities. Prospective memory is the ability to remember to do something after a delay. This thesis addresses three questions relevant to understand maintenance and execution of intentions: Is attention required to retrieve delayed intentions? What does monitoring mean in the context of prospective memory? Is prospective memory a discrete memory system or it is based on already known attentional and memory mechanisms? To answer these questions, we used electroencephalography (EEG), in (traditional) non-movement and free-movement experimental paradigms. We explored the neural substrate of prospective memory across its different stages: (1) holding intentions during a delay, (2) detecting the right context to perform the delayed intention, and (3) retrieving the content of the intention (the action to be performed). Two types of prospective memory tasks were used: Event-based prospective memory (performing a delayed intention in response to an external cue) and time-based prospective memory (performing the intention at a particular time). Results indicate that: prospective memory always requires attention, at least in experimental contexts; monitoring involves different mechanisms depending on the particular features of the prospective memory task and; prospective memory is not a discrete memory system, but relies on well-established mechanisms for attention and executive control