Plasticity in Damaged Multisensory Networks

This chapter opens by discussing functional and anatomical locations as well as neural networks of unimodal senses: vision, somatosensation, audition, gustation and olfaction. How and where these unimodal sensory systems intersect and interact with multimodal sensory processes to provide a holistic view of how experiencing complex external objects and events lead to a single multimodal percept. Reviews of current neuropsychological research on damage occurring within both unimodal and multimodal sensory networks further explain the association between these networks and how they operate together in perception. Current research reviews on cross-modal plasticity reveal the neural changes that occur in multisensory areas following brain damage and the potential benefits of this plastic reorganization of the cortex

空中ジェスチャ・インタラクションにおける触覚提示の高臨場感化および安定化に関する研究

Tohoku University昆陽雅司課

A Multi-Modal, Modified-Feedback and Self-Paced Brain-Computer Interface (BCI) to Control an Embodied Avatar's Gait

Brain-computer interfaces (BCI) have been used to control the gait of a virtual self-avatar with the aim of being used in gait rehabilitation. A BCI decodes the brain signals representing a desire to do something and transforms them into a control command for controlling external devices. The feelings described by the participants when they control a self-avatar in an immersive virtual environment (VE) demonstrate that humans can be embodied in the surrogate body of an avatar (ownership illusion). It has recently been shown that inducing the ownership illusion and then manipulating the movements of one’s self-avatar can lead to compensatory motor control strategies. In order to maximize this effect, there is a need for a method that measures and monitors embodiment levels of participants immersed in virtual reality (VR) to induce and maintain a strong ownership illusion. This is particularly true given that reaching a high level of both BCI performance and embodiment are inter-connected. To reach one of them, the second must be reached as well. Some limitations of many existing systems hinder their adoption for neurorehabilitation: 1- some use motor imagery (MI) of movements other than gait; 2- most systems allow the user to take single steps or to walk but do not allow both, which prevents users from progressing from steps to gait; 3- most of them function in a single BCI mode (cue-paced or self-paced), which prevents users from progressing from machine-dependent to machine-independent walking. Overcoming the aforementioned limitations can be done by combining different control modes and options in one single system. However, this would have a negative impact on BCI performance, therefore diminishing its usefulness as a potential rehabilitation tool. In this case, there will be a need to enhance BCI performance. For such purpose, many techniques have been used in the literature, such as providing modified feedback (whereby the presented feedback is not consistent with the user’s MI), sequential training (recalibrating the classifier as more data becomes available). This thesis was developed over 3 studies. The objective in study 1 was to investigate the possibility of measuring the level of embodiment of an immersive self-avatar, during the performing, observing and imagining of gait, using electroencephalogram (EEG) techniques, by presenting visual feedback that conflicts with the desired movement of embodied participants. The objective of study 2 was to develop and validate a BCI to control single steps and forward walking of an immersive virtual reality (VR) self-avatar, using mental imagery of these actions, in cue-paced and self-paced modes. Different performance enhancement strategies were implemented to increase BCI performance. The data of these two studies were then used in study 3 to construct a generic classifier that could eliminate offline calibration for future users and shorten training time. Twenty different healthy participants took part in studies 1 and 2. In study 1, participants wore an EEG cap and motion capture markers, with an avatar displayed in a head-mounted display (HMD) from a first-person perspective (1PP). They were cued to either perform, watch or imagine a single step forward or to initiate walking on a treadmill. For some of the trials, the avatar took a step with the contralateral limb or stopped walking before the participant stopped (modified feedback). In study 2, participants completed a 4-day sequential training to control the gait of an avatar in both BCI modes. In cue-paced mode, they were cued to imagine a single step forward, using their right or left foot, or to walk forward. In the self-paced mode, they were instructed to reach a target using the MI of multiple steps (switch control mode) or maintaining the MI of forward walking (continuous control mode). The avatar moved as a response to two calibrated regularized linear discriminant analysis (RLDA) classifiers that used the μ power spectral density (PSD) over the foot area of the motor cortex as features. The classifiers were retrained after every session. During the training, and for some of the trials, positive modified feedback was presented to half of the participants, where the avatar moved correctly regardless of the participant’s real performance. In both studies, the participants’ subjective experience was analyzed using a questionnaire. Results of study 1 show that subjective levels of embodiment correlate strongly with the power differences of the event-related synchronization (ERS) within the μ frequency band, and over the motor and pre-motor cortices between the modified and regular feedback trials. Results of study 2 show that all participants were able to operate the cued-paced BCI and the selfpaced BCI in both modes. For the cue-paced BCI, the average offline performance (classification rate) on day 1 was 67±6.1% and 86±6.1% on day 3, showing that the recalibration of the classifiers enhanced the offline performance of the BCI (p < 0.01). The average online performance was 85.9±8.4% for the modified feedback group (77-97%) versus 75% for the non-modified feedback group. For self-paced BCI, the average performance was 83% at switch control and 92% at continuous control mode, with a maximum of 12 seconds of control. Modified feedback enhanced BCI performances (p =0.001). Finally, results of study 3 show that the constructed generic models performed as well as models obtained from participant-specific offline data. The results show that there it is possible to design a participant-independent zero-training BCI.Les interfaces cerveau-ordinateur (ICO) ont été utilisées pour contrôler la marche d'un égo-avatar virtuel dans le but d'être utilisées dans la réadaptation de la marche. Une ICO décode les signaux du cerveau représentant un désir de faire produire un mouvement et les transforme en une commande de contrôle pour contrôler des appareils externes. Les sentiments décrits par les participants lorsqu'ils contrôlent un égo-avatar dans un environnement virtuel immersif démontrent que les humains peuvent être incarnés dans un corps d'un avatar (illusion de propriété). Il a été récemment démontré que provoquer l’illusion de propriété puis manipuler les mouvements de l’égo-avatar peut conduire à des stratégies de contrôle moteur compensatoire. Afin de maximiser cet effet, il existe un besoin d'une méthode qui mesure et surveille les niveaux d’incarnation des participants immergés dans la réalité virtuelle (RV) pour induire et maintenir une forte illusion de propriété. D'autre part, atteindre un niveau élevé de performances (taux de classification) ICO et d’incarnation est interconnecté. Pour atteindre l'un d'eux, le second doit également être atteint. Certaines limitations de plusieurs de ces systèmes entravent leur adoption pour la neuroréhabilitation: 1- certains utilisent l'imagerie motrice (IM) des mouvements autres que la marche; 2- la plupart des systèmes permettent à l'utilisateur de faire des pas simples ou de marcher mais pas les deux, ce qui ne permet pas à un utilisateur de passer des pas à la marche; 3- la plupart fonctionnent en un seul mode d’ICO, rythmé (cue-paced) ou auto-rythmé (self-paced). Surmonter les limitations susmentionnées peut être fait en combinant différents modes et options de commande dans un seul système. Cependant, cela aurait un impact négatif sur les performances de l’ICO, diminuant ainsi son utilité en tant qu'outil potentiel de réhabilitation. Dans ce cas, il sera nécessaire d'améliorer les performances des ICO. À cette fin, de nombreuses techniques ont été utilisées dans la littérature, telles que la rétroaction modifiée, le recalibrage du classificateur et l'utilisation d'un classificateur générique. Le projet de cette thèse a été réalisé en 3 études, avec objectif d'étudier dans l'étude 1, la possibilité de mesurer le niveau d'incarnation d'un égo-avatar immersif, lors de l'exécution, de l'observation et de l'imagination de la marche, à l'aide des techniques encéphalogramme (EEG), en présentant une rétroaction visuelle qui entre en conflit avec la commande du contrôle moteur des sujets incarnés. L'objectif de l'étude 2 était de développer un BCI pour contrôler les pas et la marche vers l’avant d'un égo-avatar dans la réalité virtuelle immersive, en utilisant l'imagerie motrice de ces actions, dans des modes rythmés et auto-rythmés. Différentes stratégies d'amélioration des performances ont été mises en œuvre pour augmenter la performance (taux de classification) de l’ICO. Les données de ces deux études ont ensuite été utilisées dans l'étude 3 pour construire des classificateurs génériques qui pourraient éliminer la calibration hors ligne pour les futurs utilisateurs et raccourcir le temps de formation. Vingt participants sains différents ont participé aux études 1 et 2. Dans l'étude 1, les participants portaient un casque EEG et des marqueurs de capture de mouvement, avec un avatar affiché dans un casque de RV du point de vue de la première personne (1PP). Ils ont été invités à performer, à regarder ou à imaginer un seul pas en avant ou la marche vers l’avant (pour quelques secondes) sur le tapis roulant. Pour certains essais, l'avatar a fait un pas avec le membre controlatéral ou a arrêté de marcher avant que le participant ne s'arrête (rétroaction modifiée). Dans l'étude 2, les participants ont participé à un entrainement séquentiel de 4 jours pour contrôler la marche d'un avatar dans les deux modes de l’ICO. En mode rythmé, ils ont imaginé un seul pas en avant, en utilisant leur pied droit ou gauche, ou la marche vers l’avant . En mode auto-rythmé, il leur a été demandé d'atteindre une cible en utilisant l'imagerie motrice (IM) de plusieurs pas (mode de contrôle intermittent) ou en maintenir l'IM de marche vers l’avant (mode de contrôle continu). L'avatar s'est déplacé en réponse à deux classificateurs ‘Regularized Linear Discriminant Analysis’ (RLDA) calibrés qui utilisaient comme caractéristiques la densité spectrale de puissance (Power Spectral Density; PSD) des bandes de fréquences µ (8-12 Hz) sur la zone du pied du cortex moteur. Les classificateurs ont été recalibrés après chaque session. Au cours de l’entrainement et pour certains des essais, une rétroaction modifiée positive a été présentée à la moitié des participants, où l'avatar s'est déplacé correctement quelle que soit la performance réelle du participant. Dans les deux études, l'expérience subjective des participants a été analysée à l'aide d'un questionnaire. Les résultats de l'étude 1 montrent que les niveaux subjectifs d’incarnation sont fortement corrélés à la différence de la puissance de la synchronisation liée à l’événement (Event-Related Synchronization; ERS) sur la bande de fréquence μ et sur le cortex moteur et prémoteur entre les essais de rétroaction modifiés et réguliers. L'étude 2 a montré que tous les participants étaient capables d’utiliser le BCI rythmé et auto-rythmé dans les deux modes. Pour le BCI rythmé, la performance hors ligne moyenne au jour 1 était de 67±6,1% et 86±6,1% au jour 3, ce qui montre que le recalibrage des classificateurs a amélioré la performance hors ligne du BCI (p <0,01). La performance en ligne moyenne était de 85,9±8,4% pour le groupe de rétroaction modifié (77-97%) contre 75% pour le groupe de rétroaction non modifié. Pour le BCI auto-rythmé, la performance moyenne était de 83% en commande de commutateur et de 92% en mode de commande continue, avec un maximum de 12 secondes de commande. Les performances de l’ICO ont été améliorées par la rétroaction modifiée (p = 0,001). Enfin, les résultats de l'étude 3 montrent que pour la classification des initialisations des pas et de la marche, il a été possible de construire des modèles génériques à partir de données hors ligne spécifiques aux participants. Les résultats montrent la possibilité de concevoir une ICO ne nécessitant aucun entraînement spécifique au participant

Exploring Virtual Reality and Doppelganger Avatars for the Treatment of Chronic Back Pain

Cognitive-behavioral models of chronic pain assume that fear of pain and subsequent avoidance behavior contribute to pain chronicity and the maintenance of chronic pain. In chronic back pain (CBP), avoidance of movements often plays a major role in pain perseverance and interference with daily life activities. In treatment, avoidance is often addressed by teaching patients to reduce pain behaviors and increase healthy behaviors. The current project explored the use of personalized virtual characters (doppelganger avatars) in virtual reality (VR), to influence motor imitation and avoidance, fear of pain and experienced pain in CBP. We developed a method to create virtual doppelgangers, to animate them with movements captured from real-world models, and to present them to participants in an immersive cave virtual environment (CAVE) as autonomous movement models for imitation. Study 1 investigated interactions between model and observer characteristics in imitation behavior of healthy participants. We tested the hypothesis that perceived affiliative characteristics of a virtual model, such as similarity to the observer and likeability, would facilitate observers’ engagement in voluntary motor imitation. In a within-subject design (N=33), participants were exposed to four virtual characters of different degrees of realism and observer similarity, ranging from an abstract stickperson to a personalized doppelganger avatar designed from 3d scans of the observer. The characters performed different trunk movements and participants were asked to imitate these. We defined functional ranges of motion (ROM) for spinal extension (bending backward, BB), lateral flexion (bending sideward, BS) and rotation in the horizontal plane (RH) based on shoulder marker trajectories as behavioral indicators of imitation. Participants’ ratings on perceived avatar appearance were recorded in an Autonomous Avatar Questionnaire (AAQ), based on an explorative factor analysis. Linear mixed effects models revealed that for lateral flexion (BS), a facilitating influence of avatar type on ROM was mediated by perceived identification with the avatar including avatar likeability, avatar-observer-similarity and other affiliative characteristics. These findings suggest that maximizing model-observer similarity may indeed be useful to stimulate observational modeling. Study 2 employed the techniques developed in study 1 with participants who suffered from CBP and extended the setup with real-world elements, creating an immersive mixed reality. The research question was whether virtual doppelgangers could modify motor behaviors, pain expectancy and pain. In a randomized controlled between-subject design, participants observed and imitated an avatar (AVA, N=17) or a videotaped model (VID, N=16) over three sessions, during which the movements BS and RH as well as a new movement (moving a beverage crate) were shown. Again, self-reports and ROMs were used as measures. The AVA group reported reduced avoidance with no significant group differences in ROM. Pain expectancy increased in AVA but not VID over the sessions. Pain and limitations did not significantly differ. We observed a moderation effect of group, with prior pain expectancy predicting pain and avoidance in the VID but not in the AVA group. This can be interpreted as an effect of personalized movement models decoupling pain behavior from movement-related fear and pain expectancy by increasing pain tolerance and task persistence. Our findings suggest that personalized virtual movement models can stimulate observational modeling in general, and that they can increase pain tolerance and persistence in chronic pain conditions. Thus, they may provide a tool for exposure and exercise treatments in cognitive behavioral treatment approaches to CBP

De animais a máquinas : humanos tecnicamente melhores nos imaginários de futuro da convergência tecnológica

Dissertação (mestrado)—Universidade de Brasília, Instituto de Ciências Sociais, Departamento de Sociologia, 2020.O tema desta investigação é discutir os imaginários sociais de ciência e tecnologia que emergem a partir da área da neuroengenharia, em sua relação com a Convergência Tecnológica de quatro disciplinas: Nanotecnologia, Biotecnologia, tecnologias da Informação e tecnologias Cognitivas - neurociências- (CT-NBIC). Estas áreas desenvolvem-se e são articuladas por meio de discursos que ressaltam o aprimoramento das capacidades físicas e cognitivas dos seres humanos, com o intuito de construir uma sociedade melhor por meio do progresso científico e tecnológico, nos limites das agendas de pesquisa e desenvolvimento (P&D). Objetivos: Os objetivos nesse cenário, são discutir as implicações éticas, econômicas, políticas e sociais deste modelo de sistema sociotécnico. Nos referimos, tanto as aplicações tecnológicas, quanto as consequências das mesmas na formação dos imaginários sociais, que tipo de relações se estabelecem e como são criadas dentro desse contexto. Conclusão: Concluímos na busca por refletir criticamente sobre as propostas de aprimoramento humano mediado pela tecnologia, que surgem enquanto parte da agenda da Convergência Tecnológica NBIC. No entanto, as propostas de melhoramento humano vão muito além de uma agenda de investigação. Há todo um quadro de referências filosóficas e políticas que defendem o aprimoramento da espécie, vertentes estas que se aliam a movimentos trans-humanistas e pós- humanistas, posições que são ao mesmo tempo éticas, políticas e econômicas. A partir de nossa análise, entendemos que ciência, tecnologia e política estão articuladas, em coprodução, em relação às expectativas de futuros que são esperados ou desejados. Ainda assim, acreditamos que há um espaço de diálogo possível, a partir do qual buscamos abrir propostas para o debate público sobre questões de ciência e tecnologia relacionadas ao aprimoramento da espécie humana.Conselho Nacional de Desenvolvimento Científico e Tecnológico (CNPq)The subject of this research is to discuss the social imaginaries of science and technology that emerge from the area of neuroengineering in relation with the Technological Convergence of four disciplines: Nanotechnology, Biotechnology, Information technologies and Cognitive technologies -neurosciences- (CT-NBIC). These areas are developed and articulated through discourses that emphasize the enhancement of human physical and cognitive capacities, the intuition it is to build a better society, through the scientific and technological progress, at the limits of the research and development (R&D) agendas. Objectives: The objective in this scenery, is to discuss the ethic, economic, politic and social implications of this model of sociotechnical system. We refer about the technological applications and the consequences of them in the formation of social imaginaries as well as the kind of social relations that are created and established in this context. Conclusion: We conclude looking for critical reflections about the proposals of human enhancement mediated by the technology. That appear as a part of the NBIC technologies agenda. Even so, the proposals of human enhancement go beyond boundaries that an investigation agenda. There is a frame of philosophical and political references that defend the enhancement of the human beings. These currents that ally to the transhumanism and posthumanism movements, positions that are ethic, politic and economic at the same time. From our analysis, we understand that science, technology and politics are articulated, are in co-production, regarding the expected and desired futures. Even so, we believe that there is a space of possible dialog, from which we look to open proposals for the public discussion on questions of science and technology related to enhancement of human beings

Interactions multisensorielles chez les musiciens

Jouer un instrument de musique demande l’interaction des informations provenant de multiples sens. Cette expérience sensorielle a des effets sur les réseaux corticaux et sur les habiletés comportementales chez les musiciens professionnels qui pratiquent pour plusieurs années. L’entrainement musical semble avoir un effet sur les sens, incluant le toucher, mais peu de recherches se sont penchées sur les habiletés tactiles chez les musiciens. L’objectif de cette thèse est d’évaluer les capacités tactiles unisensorielles et multisensorielles non musicales chez les musiciens à l’aide de méthodologies comportementales. La première étude avait pour objectif d’évaluer les temps de réaction auditifs, tactiles, et audiotactiles chez les musiciens. Les temps de réaction de 16 musiciens et 19 membres d’un groupe témoin ont été évalués. Les résultats de cette recherche suggèrent que les musiciens ont des temps de réaction significativement plus rapide pour des stimulations auditives, tactiles, et audiotactiles. La seconde étude avait comme objectif d’évaluer l’interaction d’informations audiotactiles temporelle et spectrale chez les musiciens. Les interactions audiotactiles de 13 musiciens et de 17 membres d’un groupe témoin ont été évaluées à l’aide d’illusions multisensorielles. Les résultats de cette recherche suggèrent que seulement l’interaction audiotactile temporelle est significative différente entre les groupes. La troisième étude avait pour objectif d’évaluer la localisation spatiale tactile chez les musiciens. La localisation spatiale tactile chez 17 musiciens et 20 membres d’un groupe témoin a été évaluée à l’aide de tâche de jugement d’ordre temporel tactile. Les résultats de cette recherche suggèrent que les musiciens ont un taux d’erreur plus élevé pour localiser des stimulations tactiles quand leurs bras sont croisés, mais qu’ils ont des temps de réaction plus rapides pour cette tâche. Généralement, les résultats de ces recherches suggèrent qu’un entrainement musical à long terme améliore les capacités tactiles unisensorielles et multisensorielles, mais seulement pour certaines tâches. D’autres études sont requises afin de mieux comprendre les facteurs de l’entrainement musical menant à ces changements.Playing a musical instrument requires the integration of information from multiple senses. The long-term sensory training from playing a musical instrument for many years has effects on cortical networks and behavioral abilities. Touch is a sensory modality that seems to be altered by musical training, but little research has focused on the tactile abilities of musicians. The objective of this thesis is to assess non-musical unisensory and multisensory tactile abilities in musicians using behavioral methodologies. The first study aimed at evaluating simple auditory, tactile, and audiotactile reaction times in musicians. Reaction times of 16 musicians and 19 controls were evaluated. The results of this study suggest that musicians have significantly faster response times for auditory, tactile, and audiotactile stimulations. The second study aimed at evaluating the integration of temporal and spectral audiotactile information in musicians. Audiotactile interactions of 13 musicians and 17 controls were evaluated using multisensory illusions. The results of this research suggest that only temporal audiotactile interactions are different for musicians. The third study aimed at assessing temporal tactile localization in musicians using tactile temporal order judgement task. Temporal tactile localization was evaluated in 17 musicians and 20 members of a control group. The results of this study suggest that musicians have a higher error rate to localize tactile stimulations when their arms are crossed but generally have faster reaction times for this task. All of these results suggest that musicians have altered tactile abilities. Overall, these results suggest that long-term musical training alters specific unisensory and multisensory tactile abilities. Further studies are required to better understand the factors of musical training leading to these changes and why certain interactions remain unchanged