Games and Brain-Computer Interfaces: The State of the Art

BCI gaming is a very young field; most games are proof-of-concepts. Work that compares BCIs in a game environments with traditional BCIs indicates no negative effects, or even a positive effect of the rich visual environments on the performance. The low transfer-rate of current games poses a problem for control of a game. This is often solved by changing the goal of the game. Multi-modal input with BCI forms an promising solution, as does assigning more meaningful functionality to BCI control

Rehabilitative devices for a top-down approach

In recent years, neurorehabilitation has moved from a "bottom-up" to a "top down" approach. This change has also involved the technological devices developed for motor and cognitive rehabilitation. It implies that during a task or during therapeutic exercises, new "top-down" approaches are being used to stimulate the brain in a more direct way to elicit plasticity-mediated motor re-learning. This is opposed to "Bottom up" approaches, which act at the physical level and attempt to bring about changes at the level of the central neural system. Areas covered: In the present unsystematic review, we present the most promising innovative technological devices that can effectively support rehabilitation based on a top-down approach, according to the most recent neuroscientific and neurocognitive findings. In particular, we explore if and how the use of new technological devices comprising serious exergames, virtual reality, robots, brain computer interfaces, rhythmic music and biofeedback devices might provide a top-down based approach. Expert commentary: Motor and cognitive systems are strongly harnessed in humans and thus cannot be separated in neurorehabilitation. Recently developed technologies in motor-cognitive rehabilitation might have a greater positive effect than conventional therapies

Creating Bio-adaptive Visual Cues for a Social Virtual Reality Meditation Environment

This thesis examines designing and implementing adaptive visual cues for a social virtual reality meditation environment. The system described here adapts into user’s bio- and neurofeedback and uses that data in visual cues to convey information of physiological and affective states during meditation exercises supporting two simultaneous users. The thesis shows the development process of different kinds of visual cues and attempts to pinpoint best practices, design principles and pitfalls regarding the visual cue development in this context. Also examined are the questions regarding criteria for selecting correct visual cues and how to convey information of biophysical synchronization between users. The visual cues examined here are created especially for a virtual reality environment which differs as a platform from traditional two dimensional content such as user interfaces on a computer display. Points of interests are how to embody the visual cues into the virtual reality environment so that the user experience remains immersive and the visual cues convey information correctly and in an intuitive manner

TOBE: Tangible Out-of-Body Experience

We propose a toolkit for creating Tangible Out-of-Body Experiences: exposing the inner states of users using physiological signals such as heart rate or brain activity. Tobe can take the form of a tangible avatar displaying live physiological readings to reflect on ourselves and others. Such a toolkit could be used by researchers and designers to create a multitude of potential tangible applications, including (but not limited to) educational tools about Science Technologies Engineering and Mathematics (STEM) and cognitive science, medical applications or entertainment and social experiences with one or several users or Tobes involved. Through a co-design approach, we investigated how everyday people picture their physiology and we validated the acceptability of Tobe in a scientific museum. We also give a practical example where two users relax together, with insights on how Tobe helped them to synchronize their signals and share a moment

Brain–computer interface game applications for combined neurofeedback and biofeedback treatment for children on the autism spectrum

Individuals with Autism Spectrum Disorder (ASD) show deficits in social and communicative skills, including imitation, empathy, and shared attention, as well as restricted interests and repetitive patterns of behaviors. Evidence for and against the idea that dysfunctions in the mirror neuron system are involved in imitation and could be one underlying cause for ASD is discussed in this review. Neurofeedback interventions have reduced symptoms in children with ASD by self-regulation of brain rhythms. However, cortical deficiencies are not the only cause of these symptoms. Peripheral physiological activity, such as the heart rate, is closely linked to neurophysiological signals and associated with social engagement. Therefore, a combined approach targeting the interplay between brain, body and behavior could be more effective. Brain-computer interface applications for combined neurofeedback and biofeedback treatment for children with ASD are currently nonexistent. To facilitate their use, we have designed an innovative game that includes social interactions and provides neural- and body-based feedback that corresponds directly to the underlying significance of the trained signals as well as to the behavior that is reinforced

Enhancing biofeedback-driven self-guided virtual reality exposure therapy through arousal detection from multimodal data using machine learning

Virtual reality exposure therapy (VRET) is a novel intervention technique that allows individuals to experience anxiety-evoking stimuli in a safe environment, recognise specific triggers and gradually increase their exposure to perceived threats. Public-speaking anxiety (PSA) is a prevalent form of social anxiety, characterised by stressful arousal and anxiety generated when presenting to an audience. In self-guided VRET, participants can gradually increase their tolerance to exposure and reduce anxiety-induced arousal and PSA over time. However, creating such a VR environment and determining physiological indices of anxiety-induced arousal or distress is an open challenge. Environment modelling, character creation and animation, psychological state determination and the use of machine learning (ML) models for anxiety or stress detection are equally important, and multi-disciplinary expertise is required. In this work, we have explored a series of ML models with publicly available data sets (using electroencephalogram and heart rate variability) to predict arousal states. If we can detect anxiety-induced arousal, we can trigger calming activities to allow individuals to cope with and overcome distress. Here, we discuss the means of effective selection of ML models and parameters in arousal detection. We propose a pipeline to overcome the model selection problem with different parameter settings in the context of virtual reality exposure therapy. This pipeline can be extended to other domains of interest where arousal detection is crucial. Finally, we have implemented a biofeedback framework for VRET where we successfully provided feedback as a form of heart rate and brain laterality index from our acquired multimodal data for psychological intervention to overcome anxiety

Immersive Composition for Sensory Rehabilitation: 3D Visualisation, Surround Sound, and Synthesised Music to Provoke Catharsis and Healing

There is a wide range of sensory therapies using sound, music and visual stimuli. Some focus on soothing or distracting stimuli such as natural sounds or classical music as analgesic, while other approaches emphasize the active performance of producing music as therapy. This paper proposes an immersive multi-sensory Exposure Therapy for people suffering from anxiety disorders, based on a rich, detailed surround-soundscape. This soundscape is composed to include the users’ own idiosyncratic anxiety triggers as a form of habituation, and to provoke psychological catharsis, as a non-verbal, visceral and enveloping exposure. To accurately pinpoint the most effective sounds and to optimally compose the soundscape we will monitor the participants’ physiological responses such as electroencephalography, respiration, electromyography, and heart rate during exposure. We hypothesize that such physiologically optimized sensory landscapes will aid the development of future immersive therapies for various psychological conditions, Sound is a major trigger of anxiety, and auditory hypersensitivity is an extremely problematic symptom. Exposure to stress-inducing sounds can free anxiety sufferers from entrenched avoidance behaviors, teaching physiological coping strategies and encouraging resolution of the psychological issues agitated by the sound

Exploratory psychometric validation and efficacy assessment study of social phobia treatment based on augmented and virtual reality serious games and biofeedback

Tese de mestrado integrado em Engenharia Biomédica e Biofísica (Engenharia Clínica e Instrumentação Médica), Universidade de Lisboa, Faculdade de Ciências, 2020Com o tamanho de aproximadamente dois punhos fechados e a aparência de uma enorme noz, o cérebro humano possui um poder inigualável, sendo um dos órgãos mais complexos e importantes do corpo humano. Tem a capacidade de um grande computador, capaz de receber e processar informação para depois enviar mensagens como resposta, e ainda assim é capaz de diferenciar a inteligência humana da de uma máquina – capacitando a espécie humana para muito mais, como é o caso particular dos pensamentos, sentimentos e emoções. No entanto, apesar das suas impressionantes capacidades, o cérebro também é extremamente frágil. De acordo com a Organização Mundial de Saúde, 1 em cada 4 pessoas em todo o mundo irá sofrer de uma determinada doença mental ou neurológica, nalgum ponto das suas vidas. Adicionalmente, doenças do foro mental estão entre as principais causas de incapacidade a nível mundial, visto que afetam atualmente cerca de 450 milhões de pessoas, das quais 300 milhões foram apanhadas pelo maior flagelo da mente do milénio: ansiedade e as doenças que dela derivam. Nos dias que correm, os métodos tradicionais de tratamento de patologias ansiogénicas, nomeadamente fobias, são de uma forma geral insatisfatórios, visto que tipicamente dependem de medicação, não possuem métricas objetivas nem existe uma continuidade casa/clínica. Na União Europeia, cerca de 83 milhões de pessoas sofrem de fobias. A patologia fóbica representa o grupo de perturbações de ansiedade com origem apenas, ou predominantemente, em determinadas situações bem definidas e que não representam qualquer perigo: tipicamente um medo persistente e excessivo de um objeto ou situação. Existem três tipos de fobias: fobias específicas, fobia social e agorafobia – sendo a fobia social a mais predominante. No entanto, os métodos tradicionais de tratamento, que incluem a medicação, a terapia cognitivo-comportamental e a dessensibilização sistemática, possuem bastante limitações. Por um lado, a medicação tem uma atuação relativamente rápida, porém não resolve a raiz do problema e origina tipicamente efeitos secundários fortes e prolongados no tempo. Os métodos terapêuticos, mais seguros e com maior potencial, necessitam de longos períodos de tempo para que se comecem a ver os primeiros resultados e, como tal, exigem um elevado compromisso e dedicação por parte do paciente, de tal forma que nem toda a gente é capaz de levar estes tipos de tratamento até ao fim. Adicionalmente, encontram se com frequência geograficamente distribuídos de forma assimétrica e são ainda objeto de grande estigma e tabu. Por outro lado, outra lacuna inerente à abordagem terapêutica em questão é a falta de métricas objetivas que quantifiquem o estado do paciente e o progresso do tratamento, visto que estão inteiramente dependentes da autoavaliação subjetiva da severidade do que sentem os próprios pacientes, bem como a análise, também esta subjetiva, por parte do terapeuta. Ora, este carácter subjetivo leva a uma eficácia de tratamentos muito variável, tornando estes métodos tradicionais muito pouco fiáveis. O presente trabalho de dissertação focou-se na conceptualização, implementação e teste da eficácia de um mecanismo psicoterapêutico digitalizado, que combina jogos sérios com biofeedback para melhorar o tratamento da fobia social. Jogo sério é a designação dada a experiências gamificadas desenhadas para entreter o jogador, enquanto o educam, treinam ou induzem a alteração de comportamentos, não tendo o entretenimento como objetivo central. A aplicação desta técnica no campo da saúde tem vindo a ser comprovada como uma mais valia, incentivando o envolvimento ativo e dinâmico do paciente. No que diz respeito ao biofeedback, este consiste em protocolos de condicionamento que visam manipular sistemas corporais, arbitrariamente, isto é, treino comportamental através da monitorização de processos fisiológicos específicos. Assim, a abordagem idealizada pretende colmatar as lacunas da técnica de dessensibilização sistemática, recorrendo a jogos sérios para facilitar a recriação do ambiente de exposição fóbica em qualquer lugar e de forma controlada, motivar o paciente ao longo do seu tratamento, personalizar a sua jornada através do biofeedback e, recorrendo aos sinais fisiológicos utilizados para o esse feedback, monitorizar/quantificar objetivamente o progresso do tratamento. Para tal, o estudo realizado nesta dissertação foi dividido em duas grandes fases. Primeiramente, foi concebido um método para classificação emocional, através da análise da atividade fisiológica usando Eletroencefalografia (EEG) e Fotopletismografia (PPG), experienciada aquando da visualização de 7 clips de vídeo que suscitam diferentes emoções. Para a validação desses clips, 100 sujeitos saudáveis foram submetidos à experiência e foram avaliadas as emoções sentidas, reportadas pelos próprios. Posteriormente, os vídeos mais eficazes para suscitar cada emoção foram selecionados e o protocolo repetiu-se para 25 novos sujeitos saudáveis e ambas as respostas relatadas e fisiológicas foram monitorizadas. A partir da informação recolhida, retiraram-se várias conclusões uteis para a próxima fase do estudo: relativamente à definição dos requisitos para seleção dos participantes (género, mão dominante), bem como à obtenção do biomarcador da ansiedade baseado nas métricas fisiológicas exploradas. Em conjunto com a equipa psiquiátrica do Hospital de Beja, selecionaram-se 5 doentes masculinos e destros, com idades compreendidas entre os 26 e 58 anos, para participarem no estudo, sendo divididos em dois grupos: o experimental (3) que participou num total de 8 sessões, e o de controlo (2) que não teve qualquer tipo de interação com a investigadora, sendo que a informação necessária foi transmitida pelo seu médico psiquiatra. As duas primeiras sessões dedicaram-se às apresentações e a última serviu de follow-up para avaliar o progresso do tratamento. Ao longo das 5 sessões restantes, foram aplicados os jogos sérios (com Realidade Aumentada (AR) para a terapia por exposição e com Realidade Virtual (VR) para meditação imersiva) e a avaliação emocional, através dos dados fisiológicos de EEG e PPG e do conhecimento adquirido da fase anterior, permitindo a execução de um biofeedback manual, controlando a intensidade da exposição ao estímulo fóbico. Adicionalmente, os participantes também responderam a questionários universais para avaliar os níveis de ansiedade e de fobia social, de qualidade de vida e a experiência de jogo: Beck Anxiety Inventory (BAI); Liebowitz Social Anxiety Scale (LSAS); World Health Organization Quality of Life (WHOQOL-Bref); and Game Experience Questionnaire (GEQ), respetivamente. Os resultados do grupo experimental relativamente ao grupo de controlo demonstram um decréscimo nas pontuações do BAI 6.06 vezes mais acentuado, um decréscimo nas da LSAS 4.57 vezes mais acentuado e um aumento do domínio das relações sociais da WHOQOL 4.24 vezes mais acentuado. Os jogos sérios foram validados através do GEQ, tanto o AR para estimulação fóbica gradual como VR para relaxamento e recuperação do estado desconfortável suscitado pela referida estimulação. Por fim, as métricas fisiológicas também confirmam tais conclusões, dado que se verificam diferenças significativas entre as tarefas com propósitos distintos. Em particular, o presente estudo exploratório serviu como base para testar a usabilidade e recetividade da solução inovadora, bem como aferir o seu potencial e eficácia. Contudo, os resultados devem ser considerados à luz de algumas limitações, nomeadamente no que diz respeito à amostra populacional. Próximos passos para o projeto passam pela otimização dos jogos, tanto a nível gráfico, como da robustez e autonomia dos algoritmos de biofeedback. Posteriormente, uma prova de conceito com um grupo de pacientes de dimensão significativa para se retirarem as devidas conclusões e etapas consequentes para a obtenção de comprovação/certificação clínica. Uma vez clinicamente comprovado, será possível usufruir do potencial da técnica para translação de parte da terapia para fora do ambiente clínico/hospitalar, tornando todo o processo de tratamento mais expedito e acessível. Em suma, o principal objetivo de desenvolver uma solução de jogos sérios para tratamento de fobia social, centrada no doente, foi alcançado e o estudo sugere a eficácia da aplicação da mesma.Nowadays, traditional anxiety treatment methods, namely for phobias, are overall underperforming as they typically rely on medication, and lack objective metrics and clinical/home treatment continuity. In the European Union alone, there are over 83 million people suffering from phobias, access to treatment methods is geographically asymmetrically distributed, and patients are still object of a lot of stigma and taboo. The present dissertation work focuses on conceptualizing, implementing and testing the efficacy of a digitalized psychotherapeutic approach, combining serious gaming and biofeedback for improving the treatment of one of the most common mental health disorders: social phobia. To do so, the study was divided in two major phases. First, an emotion classifying method was conceived, through the analysis of Electroencephalography (EEG) and Photoplethysmography (PPG) signal collected whilst healthy subjects were watching a set of 7 emotion-eliciting videos excerpts. To validate the videos, 100 healthy subjects underwent the experiment and self-reported the emotions perceived. Subsequently, the most effective video on eliciting each emotion was selected and the protocol was extended to 25 new healthy subjects, and both self-reported and physiological responses were monitored. From the information collected, conclusions were retrieved for the next phase: defining participant selection requirements (genre, dominant hand) as well as obtaining an anxiety biomarker based on the explored physiological metrics. Together with the psychiatric team of the Hospital de Beja, 5 male right-handed social phobia patients, aged from 26 to 58 years old, were selected to partake in the study, divided in two groups: the experimental group (3) that participated in a total of 8 sessions, and the control group (2) that had no interaction with the researcher and whose relevant information was transmitted by their physician. The first two sessions were dedicated to presentations and acquaintance with the research team, and the last served as a follow-up moment to assess the treatment progress. During the remaining 5 sessions, the Serious Gaming technique was applied (Augmented Reality (AR) to perform exposure therapy and Virtual Reality (VR) for immersive meditation) and the physiological emotion assessment, through EEG and PPG knowledge acquired in the previous phase, to perform a manual biofeedback, adapting the intensity of the exposure to the phobic stimuli. Additionally, participants also filled out self-reported questionnaires to assess anxiety and social phobia levels, quality-of-life and the gaming experience: Beck Anxiety Inventory (BAI), Liebowitz Social Anxiety Scale (LSAS), World Health Organization Quality of Life (WHOQOL-Bref) and Game Experience Questionnaire (GEQ), respectively. Results demonstrated a decrease on BAI scores 6.06 times more accentuated in the experimental against the control group, a decrease 4.57 times more accentuated on the overall LSAS traces and an increase 4.24 times more accentuated on the WHOQOL domain of social relationships. The serious games were validated through GEQ, both the AR for stimulating uncomfortable states that trigger social phobia, and the VR for inducing the relaxing state to recover from the previous phobic-stimulating experience. Lastly, the physiological features also confirm said conclusions, since there were significant differences identified between tasks with the described separate purposes. Notably, the present exploratory study served as the basis for testing the usability and receptivity of such an innovative solution, as well as assessing its potential efficacy. There is, however, the need to have an increased sample size under testing to confirm the obtained results. All in all, the primary objective of the developing a patient-oriented serious game solution for social phobia treatment was achieved and the study suggests the effectiveness of its application

Evoking Physiological Synchrony and Empathy Using Social VR with Biofeedback

With the advent of consumer grade virtual reality (VR) headsets and physiological measurement devices, new possibilities for mediated social interaction emerge enabling the immersion to environments where the visual features react to the users' physiological activation. In this study, we investigated whether and how individual and interpersonally shared biofeedback (visualised respiration rate and frontal asymmetry of electroencephalography, EEG) enhance synchrony between the users' physiological activity and perceived empathy towards the other during a compassion meditation exercise carried out in a social VR setting. The study was conducted as a laboratory experiment (N = 72) employing a Unity3D-based Dynecom immersive social meditation environment and two amplifiers to collect the psychophysiological signals for the biofeedback. The biofeedback on empathy-related EEG frontal asymmetry evoked higher self-reported empathy towards the other user than the biofeedback on respiratory activation, but the perceived empathy was highest when both feedbacks were simultaneously presented. In addition, the participants reported more empathy when there was stronger EEG frontal asymmetry synchronization between the users. The presented results inform the field of affective computing on the possibilities that VR offers for different applications of empathic technologies.Peer reviewe