Combining brain-computer interfaces and assistive technologies: state-of-the-art and challenges

In recent years, new research has brought the field of EEG-based Brain-Computer Interfacing (BCI) out of its infancy and into a phase of relative maturity through many demonstrated prototypes such as brain-controlled wheelchairs, keyboards, and computer games. With this proof-of-concept phase in the past, the time is now ripe to focus on the development of practical BCI technologies that can be brought out of the lab and into real-world applications. In particular, we focus on the prospect of improving the lives of countless disabled individuals through a combination of BCI technology with existing assistive technologies (AT). In pursuit of more practical BCIs for use outside of the lab, in this paper, we identify four application areas where disabled individuals could greatly benefit from advancements in BCI technology, namely,“Communication and Control”, “Motor Substitution”, “Entertainment”, and “Motor Recovery”. We review the current state of the art and possible future developments, while discussing the main research issues in these four areas. In particular, we expect the most progress in the development of technologies such as hybrid BCI architectures, user-machine adaptation algorithms, the exploitation of users’ mental states for BCI reliability and confidence measures, the incorporation of principles in human-computer interaction (HCI) to improve BCI usability, and the development of novel BCI technology including better EEG devices

Prefrontal cortex activation upon a demanding virtual hand-controlled task: A new frontier for neuroergonomics

open9noFunctional near-infrared spectroscopy (fNIRS) is a non-invasive vascular-based functional neuroimaging technology that can assess, simultaneously from multiple cortical areas, concentration changes in oxygenated-deoxygenated hemoglobin at the level of the cortical microcirculation blood vessels. fNIRS, with its high degree of ecological validity and its very limited requirement of physical constraints to subjects, could represent a valid tool for monitoring cortical responses in the research field of neuroergonomics. In virtual reality (VR) real situations can be replicated with greater control than those obtainable in the real world. Therefore, VR is the ideal setting where studies about neuroergonomics applications can be performed. The aim of the present study was to investigate, by a 20-channel fNIRS system, the dorsolateral/ventrolateral prefrontal cortex (DLPFC/VLPFC) in subjects while performing a demanding VR hand-controlled task (HCT). Considering the complexity of the HCT, its execution should require the attentional resources allocation and the integration of different executive functions. The HCT simulates the interaction with a real, remotely-driven, system operating in a critical environment. The hand movements were captured by a high spatial and temporal resolution 3-dimensional (3D) hand-sensing device, the LEAP motion controller, a gesture-based control interface that could be used in VR for tele-operated applications. Fifteen University students were asked to guide, with their right hand/forearm, a virtual ball (VB) over a virtual route (VROU) reproducing a 42 m narrow road including some critical points. The subjects tried to travel as long as possible without making VB fall. The distance traveled by the guided VB was 70.2 ± 37.2 m. The less skilled subjects failed several times in guiding the VB over the VROU. Nevertheless, a bilateral VLPFC activation, in response to the HCT execution, was observed in all the subjects. No correlation was found between the distance traveled by the guided VB and the corresponding cortical activation. These results confirm the suitability of fNIRS technology to objectively evaluate cortical hemodynamic changes occurring in VR environments. Future studies could give a contribution to a better understanding of the cognitive mechanisms underlying human performance either in expert or non-expert operators during the simulation of different demanding/fatiguing activities.openCarrieri, Marika; Petracca, Andrea; Lancia, Stefania; Basso Moro, Sara; Brigadoi, Sabrina; Spezialetti, Matteo; Ferrari, Marco; Placidi, Giuseppe; Quaresima, ValentinaCarrieri, Marika; Petracca, Andrea; Lancia, Stefania; BASSO MORO, Sara; Brigadoi, Sabrina; Spezialetti, Matteo; Ferrari, Marco; Placidi, Giuseppe; Quaresima, Valentin

Rehabilitative devices for a top-down approach

In recent years, neurorehabilitation has moved from a "bottom-up" to a "top down" approach. This change has also involved the technological devices developed for motor and cognitive rehabilitation. It implies that during a task or during therapeutic exercises, new "top-down" approaches are being used to stimulate the brain in a more direct way to elicit plasticity-mediated motor re-learning. This is opposed to "Bottom up" approaches, which act at the physical level and attempt to bring about changes at the level of the central neural system. Areas covered: In the present unsystematic review, we present the most promising innovative technological devices that can effectively support rehabilitation based on a top-down approach, according to the most recent neuroscientific and neurocognitive findings. In particular, we explore if and how the use of new technological devices comprising serious exergames, virtual reality, robots, brain computer interfaces, rhythmic music and biofeedback devices might provide a top-down based approach. Expert commentary: Motor and cognitive systems are strongly harnessed in humans and thus cannot be separated in neurorehabilitation. Recently developed technologies in motor-cognitive rehabilitation might have a greater positive effect than conventional therapies

An exploratory fNIRS study with immersive virtual reality: a new method for technical implementation

For over two decades Virtual Reality (VR) has been used as a useful tool in several fields, from medical and psychological treatments, to industrial and military applications. Only in recent years researchers have begun to study the neural correlates that subtend VR experiences. Even if the functional Magnetic Resonance Imaging (fMRI) is the most common and used technique, it suffers several limitations and problems. Here we present a methodology that involves the use of a new and growing brain imaging technique, functional Near-infrared Spectroscopy (fNIRS), while participants experience immersive VR. In order to allow a proper fNIRS probe application, a custom-made VR helmet was created. To test the adapted helmet, a virtual version of the line bisection task was used. Participants could bisect the lines in a virtual peripersonal or extrapersonal space, through the manipulation of a Nintendo Wiimote ® controller in order for the participants to move a virtual laser pointer. Although no neural correlates of the dissociation between peripersonal and extrapersonal space were found, a significant hemodynamic activity with respect to the baseline was present in the right parietal and occipital areas. Both advantages and disadvantages of the presented methodology are discussed

Mobile brain/body imaging of landmark-based navigation with high-density EEG.

Coupling behavioral measures and brain imaging in naturalistic, ecological conditions is key to comprehend the neural bases of spatial navigation. This highly integrative function encompasses sensorimotor, cognitive, and executive processes that jointly mediate active exploration and spatial learning. However, most neuroimaging approaches in humans are based on static, motion-constrained paradigms and they do not account for all these processes, in particular multisensory integration. Following the Mobile Brain/Body Imaging approach, we aimed to explore the cortical correlates of landmark-based navigation in actively behaving young adults, solving a Y-maze task in immersive virtual reality. EEG analysis identified a set of brain areas matching state-of-the-art brain imaging literature of landmark-based navigation. Spatial behavior in mobile conditions additionally involved sensorimotor areas related to motor execution and proprioception usually overlooked in static fMRI paradigms. Expectedly, we located a cortical source in or near the posterior cingulate, in line with the engagement of the retrosplenial complex in spatial reorientation. Consistent with its role in visuo-spatial processing and coding, we observed an alpha-power desynchronization while participants gathered visual information. We also hypothesized behavior-dependent modulations of the cortical signal during navigation. Despite finding few differences between the encoding and retrieval phases of the task, we identified transient time-frequency patterns attributed, for instance, to attentional demand, as reflected in the alpha/gamma range, or memory workload in the delta/theta range. We confirmed that combining mobile high-density EEG and biometric measures can help unravel the brain structures and the neural modulations subtending ecological landmark-based navigation

Human spatial navigation in the digital era: Effects of landmark depiction on mobile maps on navigators’ spatial learning and brain activity during assisted navigation

Navigation was an essential survival skill for our ancestors and is still a fundamental activity in our everyday lives. To stay oriented and assist navigation, our ancestors had a long history of developing and employing physical maps that communicated an enormous amount of spatial and visual information about their surroundings. Today, in the digital era, we are increasingly turning to mobile navigation devices to ease daily navigation tasks, surrendering our spatial and navigational skills to the hand-held device. On the flip side, the conveniences of such devices lead us to pay less attention to our surroundings, make fewer spatial decisions, and remember less about the surroundings we have traversed. As navigational skills and spatial memory are related to adult neurogenesis, healthy aging, education, and survival, scientists and researchers from multidisciplinary fields have made calls to develop a new account of mobile navigation assistance to preserve human navigational abilities and spatial memory. Landmarks have been advocated for special attention in developing cognitively supportive navigation systems, as landmarks are widely accepted as key features to support spatial navigation and spatial learning of an environment. Turn-by-turn direction instructions without reference to surrounding landmarks, such as those provided by most existing navigation systems, can be one of the reasons for navigators’ spatial memory deterioration during assisted navigation. Despite the benefit of landmarks in navigation and spatial learning, long-standing literature on cognitive psychology has pointed out that individuals have only a limited cognitive capacity to process presented information for a task. When the learning items exceed learners’ capacity, the performance may reach a plateau or even drop. This leads to an unexamined yet important research question on how to visualize landmarks on a mobile map to optimize navigators’ cognitive resource exertion and thus optimize their spatial learning. To investigate this question, I leveraged neuropsychological and hypothesis-driven approaches and investigated whether and how different numbers of landmarks depicted on a mobile map affected navigators’ spatial learning, cognitive load, and visuospatial encoding. Specifically, I set out a navigation experiment in three virtual urban environments, in which participants were asked to follow a given route to a specific destination with the aid of a mobile map. Three different numbers of landmarks—3, 5, and 7—along the given route were selected based on cognitive capacity literature and presented to 48 participants during map-assisted navigation. Their brain activity was recorded both during the phase of map consultation and during that of active locomotion. After navigation in each virtual city, their spatial knowledge of the traversed routes was assessed. The statistical results revealed that spatial learning improved when a medium number of landmarks (i.e., five) was depicted on a mobile map compared to the lowest evaluated number (i.e., three) of landmarks, and there was no further improvement when the highest number (i.e., seven) of landmarks were provided on the mobile map. The neural correlates that were interpreted to reflect cognitive load during map consultation increased when participants were processing seven landmarks depicted on a mobile map compared to the other two landmark conditions; by contrast, the neural correlates that indicated visuospatial encoding increased with a higher number of presented landmarks. In line with the cognitive load changes during map consultation, cognitive load during active locomotion also increased when participants were in the seven-landmark condition, compared to the other two landmark conditions. This thesis provides an exemplary paradigm to investigate navigators’ behavior and cognitive processing during map-assisted navigation and to utilize neuropsychological approaches to solve cartographic design problems. The findings contribute to a better understanding of the effects of landmark depiction (3, 5, and 7 landmarks) on navigators’ spatial learning outcomes and their cognitive processing (cognitive load and visuospatial encoding) during map-assisted navigation. Of these insights, I conclude with two main takeaways for audiences including navigation researchers and navigation system designers. First, the thesis suggests a boundary effect of the proposed benefits of landmarks in spatial learning: providing landmarks on maps benefits users’ spatial learning only to a certain extent when the number of landmarks does not increase cognitive load. Medium number (i.e., 5) of landmarks seems to be the best option in the current experiment, as five landmarks facilitate spatial learning without taxing additional cognitive resources. The second takeaway is that the increased cognitive load during map use might also spill over into the locomotion phase through the environment; thus, the locomotion phase in the environment should also be carefully considered while designing a mobile map to support navigation and environmental learning

To Drive or to Be Driven? The Impact of Autopilot, Navigation System, and Printed Maps on Driver’s Cognitive Workload and Spatial Knowledge

The technical advances in navigation systems should enhance the driving experience, supporting drivers’ spatial decision making and learning in less familiar or unfamiliar environments. Furthermore, autonomous driving systems are expected to take over navigation and driving in the near future. Yet, previous studies pointed at a still unresolved gap between environmental exploration using topographical maps and technical navigation means. Less is known about the impact of the autonomous system on the driver’s spatial learning. The present study investigates the development of spatial knowledge and cognitive workload by comparing printed maps, navigation systems, and autopilot in an unfamiliar virtual environment. Learning of a new route with printed maps was associated with a higher cognitive demand compared to the navigation system and autopilot. In contrast, driving a route by memory resulted in an increased level of cognitive workload if the route had been previously learned with the navigation system or autopilot. Way-finding performance was found to be less prone to errors when learning a route from a printed map. The exploration of the environment with the autopilot was not found to provide any compelling advantages for landmark knowledge. Our findings suggest long-term disadvantages of self-driving vehicles for spatial memory representations

EEG Correlates of Spatial Navigation in Patients with Right Hippocampal Lesion: A Mobile Brain/Body Imaging (MoBI) Study

Spatial navigation is a fundamental cognitive function that consists of different cognitive processes such as learning and decision making as well as physical locomotion. In the literature, there is a tendency to focus on cognitive elements of human spatial navigation while the presence of the body and embodied agents are neglected. Being that sensory and motor systems are integrated into the brain mechanisms according to embodied cognition theory, integrating physical movement into navigation research is crucial to investigate brain dynamics underlying human spatial navigation. Using Mobile Brain/Body Imaging (MoBI) approach, this study aims to understand electroencephalographic (EEG) activity during spatial navigation in actively moving humans. In the present study, 27 participants (9 patients with right hippocampal lesion and 18 healthy matched controls) performed a spatial navigation task in a human virtual analogue of the Morris Water Maze. Subjects were tested in both desktop and MoBI setups. In both study setups, frontal-midline (FM) theta (4-8 Hz) oscillations were examined with high-density EEG. In MoBI, EEG activity was recorded synchronously to motion capture, and the virtual environment was presented by a head-mounted display. EEG data were analyzed by using the event-related desynchronization/synchronization (ERD/ERS) method. Association between FM theta activity and spatial navigation performance was analyzed. Further, we also tested the effect of the study setup on the participant group. By comparing desktop and MoBI setups, the study aims to reveal how dynamics of the brain with hippocampal lesion change under action during spatial navigation compared to a healthy brain.Spatial navigation is a fundamental cognitive function that consists of different cognitive processes such as learning and decision making as well as physical locomotion. In the literature, there is a tendency to focus on cognitive elements of human spatial navigation while the presence of the body and embodied agents are neglected. Being that sensory and motor systems are integrated into the brain mechanisms according to embodied cognition theory, integrating physical movement into navigation research is crucial to investigate brain dynamics underlying human spatial navigation. Using Mobile Brain/Body Imaging (MoBI) approach, this study aims to understand electroencephalographic (EEG) activity during spatial navigation in actively moving humans. In the present study, 27 participants (9 patients with right hippocampal lesion and 18 healthy matched controls) performed a spatial navigation task in a human virtual analogue of the Morris Water Maze. Subjects were tested in both desktop and MoBI setups. In both study setups, frontal-midline (FM) theta (4-8 Hz) oscillations were examined with high-density EEG. In MoBI, EEG activity was recorded synchronously to motion capture, and the virtual environment was presented by a head-mounted display. EEG data were analyzed by using the event-related desynchronization/synchronization (ERD/ERS) method. Association between FM theta activity and spatial navigation performance was analyzed. Further, we also tested the effect of the study setup on the participant group. By comparing desktop and MoBI setups, the study aims to reveal how dynamics of the brain with hippocampal lesion change under action during spatial navigation compared to a healthy brain