Measuring cognitive load and cognition: metrics for technology-enhanced learning

This critical and reflective literature review examines international research published over the last decade to summarise the different kinds of measures that have been used to explore cognitive load and critiques the strengths and limitations of those focussed on the development of direct empirical approaches. Over the last 40 years, cognitive load theory has become established as one of the most successful and influential theoretical explanations of cognitive processing during learning. Despite this success, attempts to obtain direct objective measures of the theory's central theoretical construct – cognitive load – have proved elusive. This obstacle represents the most significant outstanding challenge for successfully embedding the theoretical and experimental work on cognitive load in empirical data from authentic learning situations. Progress to date on the theoretical and practical approaches to cognitive load are discussed along with the influences of individual differences on cognitive load in order to assess the prospects for the development and application of direct empirical measures of cognitive load especially in technology-rich contexts

In silico vs. Over the Clouds: On-the-Fly Mental State Estimation of Aircraft Pilots, Using a Functional Near Infrared Spectroscopy Based Passive-BCI

There is growing interest for implementing tools to monitor cognitive performance in naturalistic work and everyday life settings. The emerging field of research, known as neuroergonomics, promotes the use of wearable and portable brain monitoring sensors such as functional near infrared spectroscopy (fNIRS) to investigate cortical activity in a variety of human tasks out of the laboratory. The objective of this study was to implement an on-line passive fNIRS-based brain computer interface to discriminate two levels of working memory load during highly ecological aircraft piloting tasks. Twenty eight recruited pilots were equally split into two groups (flight simulator vs. real aircraft). In both cases, identical approaches and experimental stimuli were used (serial memorization task, consisting in repeating series of pre-recorded air traffic control instructions, easy vs. hard). The results show pilots in the real flight condition committed more errors and had higher anterior prefrontal cortex activation than pilots in the simulator, when completing cognitively demanding tasks. Nevertheless, evaluation of single trial working memory load classification showed high accuracy (>76%) across both experimental conditions. The contributions here are two-fold. First, we demonstrate the feasibility of passively monitoring cognitive load in a realistic and complex situation (live piloting of an aircraft). In addition, the differences in performance and brain activity between the two experimental conditions underscore the need for ecologically-valid investigations

Speech Processes for Brain-Computer Interfaces

Speech interfaces have become widely used and are integrated in many applications and devices. However, speech interfaces require the user to produce intelligible speech, which might be hindered by loud environments, concern to bother bystanders or the general in- ability to produce speech due to disabilities. Decoding a usera s imagined speech instead of actual speech would solve this problem. Such a Brain-Computer Interface (BCI) based on imagined speech would enable fast and natural communication without the need to actually speak out loud. These interfaces could provide a voice to otherwise mute people. This dissertation investigates BCIs based on speech processes using functional Near In- frared Spectroscopy (fNIRS) and Electrocorticography (ECoG), two brain activity imaging modalities on opposing ends of an invasiveness scale. Brain activity data have low signal- to-noise ratio and complex spatio-temporal and spectral coherence. To analyze these data, techniques from the areas of machine learning, neuroscience and Automatic Speech Recog- nition are combined in this dissertation to facilitate robust classification of detailed speech processes while simultaneously illustrating the underlying neural processes. fNIRS is an imaging modality based on cerebral blood flow. It only requires affordable hardware and can be set up within minutes in a day-to-day environment. Therefore, it is ideally suited for convenient user interfaces. However, the hemodynamic processes measured by fNIRS are slow in nature and the technology therefore offers poor temporal resolution. We investigate speech in fNIRS and demonstrate classification of speech processes for BCIs based on fNIRS. ECoG provides ideal signal properties by invasively measuring electrical potentials artifact- free directly on the brain surface. High spatial resolution and temporal resolution down to millisecond sampling provide localized information with accurate enough timing to capture the fast process underlying speech production. This dissertation presents the Brain-to- Text system, which harnesses automatic speech recognition technology to decode a textual representation of continuous speech from ECoG. This could allow to compose messages or to issue commands through a BCI. While the decoding of a textual representation is unparalleled for device control and typing, direct communication is even more natural if the full expressive power of speech - including emphasis and prosody - could be provided. For this purpose, a second system is presented, which directly synthesizes neural signals into audible speech, which could enable conversation with friends and family through a BCI. Up to now, both systems, the Brain-to-Text and synthesis system are operating on audibly produced speech. To bridge the gap to the final frontier of neural prostheses based on imagined speech processes, we investigate the differences between audibly produced and imagined speech and present first results towards BCI from imagined speech processes. This dissertation demonstrates the usage of speech processes as a paradigm for BCI for the first time. Speech processes offer a fast and natural interaction paradigm which will help patients and healthy users alike to communicate with computers and with friends and family efficiently through BCIs

Using near infrared spectroscopy and heart rate variability to detect mental overload

Mental workload is a key factor influencing the occurrence of human error, especially during piloting and remotely operated vehicle (ROV) operations, where safety depends on the ability of pilots to act appropriately. In particular, excessively high or low mental workload can lead operators to neglect critical information. The objective of the present study is to investigate the potential of functional Near Infrared Spectroscopy (fNIRS) – a non-invasive method of measuring prefrontal cortex activity – in combination with measurements of heart rate variability (HRV), to predict mental workload during a simulated piloting task, with particular regard to task engagement and disengagement. Twelve volunteers performed a computer-based piloting task in which they were asked to follow a dynamic target with their aircraft, a task designed to replicate key cognitive demands associated with real life ROV operating tasks. In order to cover a wide range of mental workload levels, task difficulty was manipulated in terms of processing load and difficulty of control – two critical sources of workload associated with piloting and remotely operating a vehicle. Results show that both fNIRS and HRV are sensitive to different levels of mental workload; notably, lower prefrontal activation as well as a lower LF/HF ratio at the highest level of difficulty, suggest that these measures are suitable for mental overload detection. Moreover, these latter measurements point towards the existence of a quadratic model of mental workload

Determining Shared Working Memory Systems for Rhythmic Incongruities in Music and Language using functional Near-Infrared Spectroscopy

Rhythmic organization of auditory information is used differently in the retention of music and spoken language. However, similar areas of the prefrontal cortex (PFC) have been implicated in the retention of unusual rhythmic patterns. This study investigated the degree of PFC activation using functional near-infrared spectroscopy (fNIRS) during three rhythmic pattern manipulation working memory tasks. In addition the normalized pair-wise variability index (NPVI) was tested as a measure of rhythmic accuracy. Of the six participants considered, three demonstrated greater activation of the right PFC in response to the Rhythmic Motor task, a manipulation of musical rhythms. Similar activation was observed for the Stress Speech task, which altered stress patterns in natural speech. No changes in activation were observed in the Rhythmic Speech task, which paired speech with metric patterns. The NPVI values did not reflect task performance. Refinement is needed to determine if the current procedure accurately measures rhythmic working memory

Brain activity underlying the recovery of meaning from degraded speech: a functional near-infrared spectroscopy (fNIRS) study

The purpose of this study was to establish whether functional near-infrared spectroscopy (fNIRS), an emerging brain-imaging technique based on optical principles, is suitable for studying the brain activity that underlies effortful listening. In an event-related fNIRS experiment, normally-hearing adults listened to sentences that were either clear or degraded (noise vocoded). These sentences were presented simultaneously with a non-speech distractor, and on each trial participants were instructed to attend either to the speech or to the distractor. The primary region of interest for the fNIRS measurements was the left inferior frontal gyrus (LIFG), a cortical region involved in higher-order language processing. The fNIRS results confirmed findings previously reported in the functional magnetic resonance imaging (fMRI) literature. Firstly, the LIFG exhibited an elevated response to degraded versus clear speech, but only when attention was directed towards the speech. This attention-dependent increase in frontal brain activation may be a neural marker for effortful listening. Secondly, during attentive listening to degraded speech, the haemodynamic response peaked significantly later in the LIFG than in superior temporal cortex, possibly reflecting the engagement of working memory to help reconstruct the meaning of degraded sentences. The homologous region in the right hemisphere may play an equivalent role to the LIFG in some left-handed individuals. In conclusion, fNIRS holds promise as a flexible tool to examine the neural signature of effortful listening

Brain regions and functional interactions supporting early word recognition in the face of input variability

Perception and cognition in infants have been traditionally investigated using habituation paradigms, assuming that babies' memories in laboratory contexts are best constructed after numerous repetitions of the very same stimulus in the absence of interference. A crucial, yet open, question regards how babies deal with stimuli experienced in a fashion similar to everyday learning situations-namely, in the presence of interfering stimuli. To address this question, we used functional near-infrared spectroscopy to test 40 healthy newborns on their ability to encode words presented in concomitance with other words. The results evidenced a habituation-like hemodynamic response during encoding in the left-frontal region, which was associated with a progressive decrement of the functional connections between this region and the left-temporal, right-temporal, and right-parietal regions. In a recognition test phase, a characteristic neural signature of recognition recruited first the right-frontal region and subsequently the right-parietal ones. Connections originating from the right-temporal regions to these areas emerged when newborns listened to the familiar word in the test phase. These findings suggest a neural specialization at birth characterized by the lateralization of memory functions: the interplay between temporal and left-frontal regions during encoding and between temporo-parietal and right-frontal regions during recognition of speech sounds. Most critically, the results show that newborns are capable of retaining the sound of specific words despite hearing other stimuli during encoding. Thus, habituation designs that include various items may be as effective for studying early memory as repeated presentation of a single word.European Research Council under European Union 269502 CONICYT-Chile Program PIA/BASAL FB0003 "Progetto strategico NEURAT" from the University of Padua CONICYT-Chile Program PAI/Academia 7913002

Validation of fNIRS System as a Technique to Monitor Cognitive Workload

CognitiveWorkload (CW) is a key factor in the human learning context. Knowing the optimal amount of CW is essential to maximise cognitive performance, emerging as an important variable in e-learning systems and Brain-Computer Interfaces (BCI) applications. Functional Near-Infrared Spectroscopy (fNIRS) has emerged as a promising avenue of brain discovery because of its easy setup and robust results. It is, in fact, along with Electroencephalography (EEG), an encouraging technique in the context of BCI. Brain- Computer Interfaces, by tracking the user’s cognitive state, are suitable for educational systems. Thus, this work sought to validate the fNIRS technique for monitoring different CW stages. For this purpose, we acquired the fNIRS and EEG signals when performing cognitive tasks, which included a progressive increase of difficulty and simulation of the learning process. We also used the breathing sensor and the participants’ facial expressions to assess their cognitive status. We found that both visual inspections of fNIRS signals and power spectral analysis of EEG bands are not sufficient for discriminating cognitive states, nor quantify CW. However, by applying machine learning (ML) algorithms, we were able to distinguish these states with mean accuracies of 79.8%, reaching a value of 100% in one specific case. Our findings provide evidence that fNIRS technique has the potential to monitor different levels of CW. Furthermore, our results suggest that this technique allied with the EEG and combined via ML algorithms is a promising tool to be used in the e-learning and BCI fields for its skill to discriminate and characterize cognitive states.O esforço cognitivo (CW) é um factor relevante no contexto da aprendizagem humana. Conhecer a quantidade óptima de CW é essencial para maximizar o desempenho cognitivo, surgindo como uma variável importante em sistemas de e-learning e aplicações de Interfaces Cérebro-Computador (BCI). A Espectroscopia Funcional de Infravermelho Próximo (fNIRS) emergiu como uma via de descoberta do cérebro devido à sua fácil configuração e resultados robustos. É, de facto, juntamente com a Electroencefalografia (EEG), uma técnica encorajadora no contexto de BCI. As interfaces cérebro-computador, ao monitorizar o estado cognitivo do utilizador, são adequadas para sistemas educativos. Assim, este trabalho procurou validar o sistema de fNIRS como uma técnica de monitorização de CW. Para este efeito, adquirimos os sinais fNIRS e EEG aquando da execução de tarefas cognitivas, que incluiram um aumento progressivo de dificuldade e simulação do processo de aprendizagem. Utilizámos, ainda, o sensor de respiração e as expressões faciais dos participantes para avaliar o seu estado cognitivo. Verificámos que tanto a inspeção visual dos sinais de fNIRS como a análise espectral dos sinais de EEG não são suficientes para discriminar estados cognitivos, nem para quantificar o CW. No entanto, aplicando algoritmos de machine learning (ML), fomos capazes de distinguir estes estados com exatidões médias de 79.8%, chegando a atingir o valor de 100% num caso específico. Os nossos resultados fornecem provas da prospecção da técnica fNIRS para supervisionar diferentes níveis de CW. Além disso, os nossos resultados sugerem que esta técnica aliada à de EEG e combinada via algoritmos ML é uma ferramenta promissora a ser utilizada nos campos do e-learning e de BCI, pela sua capacidade de discriminar e caracterizar estados cognitivos

Aging effects on prefrontal cortex oxygenation in a posture-cognition dual-task: an fNIRS pilot study

The aging process alters upright posture and locomotion control from an automatically processed to a more cortically controlled one. The present study investigated a postural-cognitive dual-task paradigm in young and older adults using functional Near- Infrared Spectroscopy (fNIRS).Methods: Twenty healthy participants (10 older adults 72 ± 3 y, 10 young adults 23 ± 3 y) performed a cognitive (serial subtractions) and a postural task (tandem stance) as single-tasks (ST) and concurrently as a dual-task (DT) while the oxygenation levels of the dorsolateral prefrontal cortex (DLPFC) were measured.Results: In the cognitive task, young adults performed better than older adults in both conditions (ST and DT) and could further increase the number of correct answers from ST to DT (all ps ≤ 0.027) while no change was found for older adults. No significant effects were found for the postural performance. Cerebral oxygenation values (O2Hb) increased significantly from baseline to the postural ST (p = 0.033), and from baseline to the DT (p = 0.031) whereas no changes were found in deoxygenated hemoglobin (HHb). Finally, the perceived exertion differed between all conditions (p ≤  0.003) except for the postural ST and the DT (p = 0.204).Conclusions: There was a general lack of age-related changes except the better cognitive performance under motor-cognitive conditions in young compared to older adults. However, the current results point out that DLPFC is influenced more strongly by postural than cognitive load. Future studies should assess the different modalities of cognitive as well as postural load