Cybathlon experiences of the Graz BCI racing team Mirage91 in the brain-computer interface discipline

Abstract Background In this work, we share our experiences made at the world-wide first CYBATHLON, an event organized by the Eidgenössische Technische Hochschule Zürich (ETH Zürich), which took place in Zurich in October 2016. It is a championship for severely motor impaired people using assistive prototype devices to compete against each other. Our team, the Graz BCI Racing Team MIRAGE91 from Graz University of Technology, participated in the discipline “Brain-Computer Interface Race”. A brain-computer interface (BCI) is a device facilitating control of applications via the user’s thoughts. Prominent applications include assistive technology such as wheelchairs, neuroprostheses or communication devices. In the CYBATHLON BCI Race, pilots compete in a BCI-controlled computer game. Methods We report on setting up our team, the BCI customization to our pilot including long term training and the final BCI system. Furthermore, we describe CYBATHLON participation and analyze our CYBATHLON result. Results We found that our pilot was compliant over the whole time and that we could significantly reduce the average runtime between start and finish from initially 178 s to 143 s. After the release of the final championship specifications with shorter track length, the average runtime converged to 120 s. We successfully participated in the qualification race at CYBATHLON 2016, but performed notably worse than during training, with a runtime of 196 s. Discussion We speculate that shifts in the features, due to the nonstationarities in the electroencephalogram (EEG), but also arousal are possible reasons for the unexpected result. Potential counteracting measures are discussed. Conclusions The CYBATHLON 2016 was a great opportunity for our student team. We consolidated our theoretical knowledge and turned it into practice, allowing our pilot to play a computer game. However, further research is required to make BCI technology invariant to non-task related changes of the EEG

Competing at the Cybathlon championship for people with disabilities: Long-term motor imagery brain-computer interface training of a cybathlete who has tetraplegia

BACKGROUND: The brain–computer interface (BCI) race at the Cybathlon championship, for people with disabilities, challenges teams (BCI researchers, developers and pilots with spinal cord injury) to control an avatar on a virtual racetrack without movement. Here we describe the training regime and results of the Ulster University BCI Team pilot who has tetraplegia and was trained to use an electroencephalography (EEG)-based BCI intermittently over 10 years, to compete in three Cybathlon events. METHODS: A multi-class, multiple binary classifier framework was used to decode three kinesthetically imagined movements (motor imagery of left arm, right arm, and feet), and relaxed state. Three game paradigms were used for training i.e., NeuroSensi, Triad, and Cybathlon Race: BrainDriver. An evaluation of the pilot’s performance is presented for two Cybathlon competition training periods—spanning 20 sessions over 5 weeks prior to the 2019 competition, and 25 sessions over 5 weeks in the run up to the 2020 competition. RESULTS: Having participated in BCI training in 2009 and competed in Cybathlon 2016, the experienced pilot achieved high two-class accuracy on all class pairs when training began in 2019 (decoding accuracy > 90%, resulting in efficient NeuroSensi and Triad game control). The BrainDriver performance (i.e., Cybathlon race completion time) improved significantly during the training period, leading up to the competition day, ranging from 274–156 s (255 ± 24 s to 191 ± 14 s mean ± std), over 17 days (10 sessions) in 2019, and from 230–168 s (214 ± 14 s to 181 ± 4 s), over 18 days (13 sessions) in 2020. However, on both competition occasions, towards the race date, the performance deteriorated significantly. CONCLUSIONS: The training regime and framework applied were highly effective in achieving competitive race completion times. The BCI framework did not cope with significant deviation in electroencephalography (EEG) observed in the sessions occurring shortly before and during the race day. Changes in cognitive state as a result of stress, arousal level, and fatigue, associated with the competition challenge and performance pressure, were likely contributing factors to the non-stationary effects that resulted in the BCI and pilot achieving suboptimal performance on race day. Trial registration not registered SUPPLEMENTARY INFORMATION: The online version contains supplementary material available at 10.1186/s12984-022-01073-9

The Cybathlon BCI race: Successful longitudinal mutual learning with two tetraplegic users

This work aims at corroborating the importance and efficacy of mutual learning in motor imagery (MI) brain–computer interface (BCI) by leveraging the insights obtained through our participation in the BCI race of the Cybathlon event. We hypothesized that, contrary to the popular trend of focusing mostly on the machine learning aspects of MI BCI training, a comprehensive mutual learning methodology that reinstates the three learning pillars (at the machine, subject, and application level) as equally significant could lead to a BCI–user symbiotic system able to succeed in real-world scenarios such as the Cybathlon event. Two severely impaired participants with chronic spinal cord injury (SCI), were trained following our mutual learning approach to control their avatar in a virtual BCI race game. The competition outcomes substantiate the effectiveness of this type of training. Most importantly, the present study is one among very few to provide multifaceted evidence on the efficacy of subject learning during BCI training. Learning correlates could be derived at all levels of the interface—application, BCI output, and electroencephalography (EEG) neuroimaging—with two end-users, sufficiently longitudinal evaluation, and, importantly, under real-world and even adverse conditions

Long multi-stage training for a motor-impaired user in a BCI competition

International audienceIn a Mental Imagery Brain-Computer Interface the user has to perform a specific mental task that generates electroencephalography (EEG) components, which can be translated in commands to control a BCI system. The development of a high-performance MI-BCI requires a long training, lasting several weeks or months, in order to improve the ability of the user to manage his/her mental tasks. This works aims to present the design of a MI-BCI combining mental imaginary and cognitive tasks for a severely motor impaired user, involved in the BCI race of the Cybathlon event, a competition of people with severe motor disability. In the BCI-race, the user becomes a pilot in a virtual race game against up to three other pilots, in which each pilot has to control his/her virtual car by his/her mental tasks. We present all the procedures followed to realize an effective MI-BCI, from the user's first contact with a BCI technology to actually controlling a video-game through her EEG. We defined a multi-stage user-centered training protocol in order to successfully control a BCI, even in a stressful situation, such as that of a competition. We put a specific focus on the human aspects that influenced the long training phase of the system and the participation to the competition

Novel Bidirectional Body - Machine Interface to Control Upper Limb Prosthesis

Objective. The journey of a bionic prosthetic user is characterized by the opportunities and limitations involved in adopting a device (the prosthesis) that should enable activities of daily living (ADL). Within this context, experiencing a bionic hand as a functional (and, possibly, embodied) limb constitutes the premise for mitigating the risk of its abandonment through the continuous use of the device. To achieve such a result, different aspects must be considered for making the artificial limb an effective support for carrying out ADLs. Among them, intuitive and robust control is fundamental to improving amputees’ quality of life using upper limb prostheses. Still, as artificial proprioception is essential to perceive the prosthesis movement without constant visual attention, a good control framework may not be enough to restore practical functionality to the limb. To overcome this, bidirectional communication between the user and the prosthesis has been recently introduced and is a requirement of utmost importance in developing prosthetic hands. Indeed, closing the control loop between the user and a prosthesis by providing artificial sensory feedback is a fundamental step towards the complete restoration of the lost sensory-motor functions. Within my PhD work, I proposed the development of a more controllable and sensitive human-like hand prosthesis, i.e., the Hannes prosthetic hand, to improve its usability and effectiveness. Approach. To achieve the objectives of this thesis work, I developed a modular and scalable software and firmware architecture to control the Hannes prosthetic multi-Degree of Freedom (DoF) system and to fit all users’ needs (hand aperture, wrist rotation, and wrist flexion in different combinations). On top of this, I developed several Pattern Recognition (PR) algorithms to translate electromyographic (EMG) activity into complex movements. However, stability and repeatability were still unmet requirements in multi-DoF upper limb systems; hence, I started by investigating different strategies to produce a more robust control. To do this, EMG signals were collected from trans-radial amputees using an array of up to six sensors placed over the skin. Secondly, I developed a vibrotactile system to implement haptic feedback to restore proprioception and create a bidirectional connection between the user and the prosthesis. Similarly, I implemented an object stiffness detection to restore tactile sensation able to connect the user with the external word. This closed-loop control between EMG and vibration feedback is essential to implementing a Bidirectional Body - Machine Interface to impact amputees’ daily life strongly. For each of these three activities: (i) implementation of robust pattern recognition control algorithms, (ii) restoration of proprioception, and (iii) restoration of the feeling of the grasped object's stiffness, I performed a study where data from healthy subjects and amputees was collected, in order to demonstrate the efficacy and usability of my implementations. In each study, I evaluated both the algorithms and the subjects’ ability to use the prosthesis by means of the F1Score parameter (offline) and the Target Achievement Control test-TAC (online). With this test, I analyzed the error rate, path efficiency, and time efficiency in completing different tasks. Main results. Among the several tested methods for Pattern Recognition, the Non-Linear Logistic Regression (NLR) resulted to be the best algorithm in terms of F1Score (99%, robustness), whereas the minimum number of electrodes needed for its functioning was determined to be 4 in the conducted offline analyses. Further, I demonstrated that its low computational burden allowed its implementation and integration on a microcontroller running at a sampling frequency of 300Hz (efficiency). Finally, the online implementation allowed the subject to simultaneously control the Hannes prosthesis DoFs, in a bioinspired and human-like way. In addition, I performed further tests with the same NLR-based control by endowing it with closed-loop proprioceptive feedback. In this scenario, the results achieved during the TAC test obtained an error rate of 15% and a path efficiency of 60% in experiments where no sources of information were available (no visual and no audio feedback). Such results demonstrated an improvement in the controllability of the system with an impact on user experience. Significance. The obtained results confirmed the hypothesis of improving robustness and efficiency of a prosthetic control thanks to of the implemented closed-loop approach. The bidirectional communication between the user and the prosthesis is capable to restore the loss of sensory functionality, with promising implications on direct translation in the clinical practice

Estratégias de controle de trajetórias para cadeira de rodas robotizadas

Orientador: Eleri CardozoDissertação (mestrado) - Universidade Estadual de Campinas, Faculdade de Engenharia Elétrica e de ComputaçãoResumo: Desde os anos 80, diversos trabalhos foram publicados com o objetivo de propor soluções alternativas para usuários de cadeira de rodas motorizadas com severa deficiência motora e que não possuam capacidade de operar um joystick mecânico. Dentre essas soluções estão interfaces assistivas que auxiliam no comando da cadeira de rodas através de diversos mecanismos como expressões faciais, interfaces cérebro-computador, e rastreamento de olho. Além disso, as cadeiras de rodas ganharam certa autonomia para realizar determinadas tarefas que vão de desviar de obstáculos, abrir portas e até planejar e executar rotas. Para que estas tarefas possam ser executadas, é necessário que as cadeiras de rodas tenham estruturas não convencionais, habilidade de sensoriamento do ambiente e estratégias de controle de locomoção. O objetivo principal é disponibilizar uma cadeira de rodas que ofereça conforto ao usuário e que possua um condução segura não importando o tipo de deficiência do usuário. Entretanto, durante a condução da cadeira de rodas, o desalinhamento das rodas castores podem oferecer certo perigo ao usuário, uma vez que, dependendo da maneira em que elas estejam orientadas, instabilidades podem ocorrer, culminando em acidentes. Da mesma forma, o desalinhamento das rodas castores é considerado um dos principais causadores de desvios de trajetória que ocorrem durante a movimentação da cadeira de rodas, juntamente com diferentes distribuições de pesos ou diferentes atritos entre as rodas e o chão. Nesta dissertação, é considerado apenas o desalinhamento das rodas castores como único causador de desvio de trajetória da cadeira de rodas e, dessa forma, são propostas soluções que possam reduzir ou até mesmo eliminar o efeito deste desalinhamento. Com a implementação das melhores soluções desenvolvidas neste trabalho, é possível fazer com que diversas interfaces assistivas que têm baixa taxa de comandos possam ser utilizadas, uma vez que o usuário não precisa, constantemente, corrigir o desvio da trajetória desejada. Ademais, é elaborado um novo projeto de cadeira de rodas "inteligente" para a implementação das técnicas desenvolvidas neste trabalhoAbstract: Since the 1980s several works were published proposing alternative solutions for users of powered wheelchairs with severe mobility impairments and that are not able to operate a mechanical joystick. Such solutions commonly focus on assistive interfaces that help commanding the wheelchair through distinct mechanisms such as facial expressions, brain-computer interfaces, and eye tracking. Besides that, the wheelchairs have achieved a certain level of autonomy to accomplish determined tasks such as obstacle avoidance, doors opening and even path planning and execution. For these tasks to be performed, it is necessary the wheelchairs to have a non conventional designs, ability to sense the environment and locomotion control strategies. The ultimate objective is to offer a comfortable and safe conduction no matter the user's mobility impairments. However, while driving the wheelchair, the caster wheels' misalignment might offer risks to the user, because, depending on the way they are initially oriented, instabilities may occur causing accidents. Similarly, the caster wheels' misalignment can be considered, among others like different weight distribution or different friction between wheel and floor, one of the main causes of path deviation from the intended trajectory while the wheelchair is moving. In this dissertation, it is considered the caster wheels' misalignment as the unique generator of wheelchair path deviation and, therefore, it is proposed different solutions in order to reduce or even eliminate the effects of the misalignment. The implementation of the best solutions developed in this work allows assistive interfaces with low rate of commands to be widespread, once the user does not need to, constantly, correct path deviation. Additionally, a new smart wheelchair project is elaborated for the implementation of the techniques developed in this workMestradoEngenharia de ComputaçãoMestre em Engenharia Elétrica88882.329382/2019-01CAPE

Human Enhancement Technologies and Our Merger with Machines

A cross-disciplinary approach is offered to consider the challenge of emerging technologies designed to enhance human bodies and minds. Perspectives from philosophy, ethics, law, and policy are applied to a wide variety of enhancements, including integration of technology within human bodies, as well as genetic, biological, and pharmacological modifications. Humans may be permanently or temporarily enhanced with artificial parts by manipulating (or reprogramming) human DNA and through other enhancement techniques (and combinations thereof). We are on the cusp of significantly modifying (and perhaps improving) the human ecosystem. This evolution necessitates a continuing effort to re-evaluate current laws and, if appropriate, to modify such laws or develop new laws that address enhancement technology. A legal, ethical, and policy response to current and future human enhancements should strive to protect the rights of all involved and to recognize the responsibilities of humans to other conscious and living beings, regardless of what they look like or what abilities they have (or lack). A potential ethical approach is outlined in which rights and responsibilities should be respected even if enhanced humans are perceived by non-enhanced (or less-enhanced) humans as “no longer human” at all

Towards a Simulator Tool for Predicting Sprinting and Long Jump Motions with and without Running-Specific Prostheses: An Optimization-Based Approach

The performances of sprinters and long jumpers with below the knee amputation (BKA) have improved continuously since the development of prostheses specifically for athletic movements. In the last years, a number of athletes with BKA have attempted to compete in non-amputee competitions. Due to the specific shape and material properties of the running-specific prosthesis (RSP), concerns exist that it may give athletes an advantage over non-amputee athletes. In this work, we investigate and compare sprinting and long jump movements of athletes with and without unilateral BKA using accurate computer models. In this context, the aim of the work is to describe similarities and differences between the athletes’ movements and to show that the employed model- and optimization-based computations are useful for this purpose. We created subject-specific multi-body models for five different athletes (four non-amputee athletes, one athlete with unilateral BKA) in order to be able to investigate the different movements. Depending on the research question, the models vary in the number of degrees of freedom (DOFs), from 16 DOFs for a two-dimensional model in the sagittal plane to 31 DOFs for a three-dimensional model. For the athlete with BKA, we created a three-segment model of the RSP with one rotational DOF in the sagittal plane. The respective motion is described by a sequence of several phases, which differ by the type of ground contact. Each of these phases is described by its own set of ordinary differential equations (ODEs) or differential algebraic equations (DAEs). We use multi-phase optimal control problems (OCPs) with discontinuities to generate sprint and long jump motions. Three different formulations of OCPs are adopted in this work. (1) We formulate a least squares OCP to reconstruct the dynamics of sprint and long jump motion capture recordings of the individual athletes. (2) For the generation of realistic motions, which can be used for prediction, we formulate a synthesis OCP; this optimizes an objective function consisting of a weighted combination of chosen optimization criteria. (3) Last, in the study of sprint movements, we use an inverse optimal control problem (IOCP): this consists of an inner loop, in which a synthesis OCP is solved, and an outer loop, which adjusts the weights of the individual optimization criteria such that the distance between the inner loop solution and a reference movement becomes minimal. We have successfully applied these three optimization problem formulations to the computation of two sprint steps of three athletes without and one athlete with unilateral transtibial amputation. Here, the movements of the non-amputee athletes differ from that of the amputee athlete in a large number of variables. In particular, the athletes use different actuation strategies for running with and without a RSP. We have observed lower torques in the amputee athlete in the leg affected by the amputation than in the non-amputee control group. In contrast, significantly larger torques occurred in the joints of the upper extremity in the amputee athlete. Furthermore, the comparison has shown that the asymmetry created by the RSP is reflected throughout the body and affects the entire movement. Using the OCPs for motion reconstruction (1) and synthesis (2), we have successfully computed the last three steps of the approach and the jump of a long jump for an athlete without and an athlete with unilateral amputation. In the reconstructed solutions, the amputee athlete achieves a greater jump distance compared to the non-amputee athlete, despite a slower approach velocity, because his take-off is more efficient. In the synthesis solutions, on the other hand, the non-amputee athlete achieves the greater jump distance because he generates a greater vertical force during the take-off and achieves a better ratio of gain of vertical to loss of horizontal velocity. Finally, we have presented our idea of a simulator tool to compare the amputee athlete with himself without amputation and have demonstrated it using the sprint and long jump movements. For this purpose, we have kept the model of the athlete with unilateral transtibial amputation from the previous studies and have created a non-amputee version of the same model by mirroring the biological leg. We have selected one objective function each for sprinting and for long jump and have solved the OCP for motion synthesis (2) for both model versions. Using the differences to the solutions based on the models of two real athletes, we have highlighted the importance of the simulator tool in the evaluation of advantages and disadvantages due to the use of the RSP