Emerging ExG-based NUI Inputs in Extended Realities : A Bottom-up Survey

Incremental and quantitative improvements of two-way interactions with extended realities (XR) are contributing toward a qualitative leap into a state of XR ecosystems being efficient, user-friendly, and widely adopted. However, there are multiple barriers on the way toward the omnipresence of XR; among them are the following: computational and power limitations of portable hardware, social acceptance of novel interaction protocols, and usability and efficiency of interfaces. In this article, we overview and analyse novel natural user interfaces based on sensing electrical bio-signals that can be leveraged to tackle the challenges of XR input interactions. Electroencephalography-based brain-machine interfaces that enable thought-only hands-free interaction, myoelectric input methods that track body gestures employing electromyography, and gaze-tracking electrooculography input interfaces are the examples of electrical bio-signal sensing technologies united under a collective concept of ExG. ExG signal acquisition modalities provide a way to interact with computing systems using natural intuitive actions enriching interactions with XR. This survey will provide a bottom-up overview starting from (i) underlying biological aspects and signal acquisition techniques, (ii) ExG hardware solutions, (iii) ExG-enabled applications, (iv) discussion on social acceptance of such applications and technologies, as well as (v) research challenges, application directions, and open problems; evidencing the benefits that ExG-based Natural User Interfaces inputs can introduceto the areaof XR.Peer reviewe

Emerging ExG-based NUI Inputs in Extended Realities : A Bottom-up Survey

Incremental and quantitative improvements of two-way interactions with extended realities (XR) are contributing toward a qualitative leap into a state of XR ecosystems being efficient, user-friendly, and widely adopted. However, there are multiple barriers on the way toward the omnipresence of XR; among them are the following: computational and power limitations of portable hardware, social acceptance of novel interaction protocols, and usability and efficiency of interfaces. In this article, we overview and analyse novel natural user interfaces based on sensing electrical bio-signals that can be leveraged to tackle the challenges of XR input interactions. Electroencephalography-based brain-machine interfaces that enable thought-only hands-free interaction, myoelectric input methods that track body gestures employing electromyography, and gaze-tracking electrooculography input interfaces are the examples of electrical bio-signal sensing technologies united under a collective concept of ExG. ExG signal acquisition modalities provide a way to interact with computing systems using natural intuitive actions enriching interactions with XR. This survey will provide a bottom-up overview starting from (i) underlying biological aspects and signal acquisition techniques, (ii) ExG hardware solutions, (iii) ExG-enabled applications, (iv) discussion on social acceptance of such applications and technologies, as well as (v) research challenges, application directions, and open problems; evidencing the benefits that ExG-based Natural User Interfaces inputs can introduceto the areaof XR.Peer reviewe

Advanced Augmentative and Alternative Communication System Based in Physiological Control

Dyskinetic Cerebral Palsy (DCP) is mainly characterized by alterations in muscle tone and involuntary movements. Therefore, these people present with difficulties in coordination and movement control, which makes walking difficult and affects their posture when seated. Additionally, their cognitive performance varies between being completely normal and severe mental retardation. People with DCP were selected as the objective of this thesis due to their multiple and complex limitations (speech problems and motor control) and because their capabilities have a great margin for improvement thanks to physiological control systems. Given their communication difficulties, some people with DCP have good motor con-trol and can communicate with written language. However, most have difficulty using Augmentative and Alternative Communication (AAC) systems. People with DCP gen-erally use concept boards to indicate the idea they want to communicate. However, most communication solutions available today are based on proprietary software that makes it difficult to customize the concept board and this type of control system. This is the motivation behind this thesis, with the aim of creating an interface with characteristics, able to be adapted to the user needs and limitations. Thus, this thesis proposes an Augmentative and Alternative Communication System for people with DCP based on physiological control. In addition, an innovative system for direct con-trol of concept boards with EMG is proposed. This control system is based on a physi-cal model that reproduces the muscular mechanical response (stiffness, inertia and viscosity). It allows for a selection of elements thanks to small pulses of EMG signal with sensors on a muscle with motor control. Its main advantage is the possibility of correcting errors during selection associated with uncontrolled muscle impulses, avoid-ing sustained muscle effort and thus reduced fatigue.La Parálisis Cerebral de tipo Discinésica (DCP) se caracteriza principalmente por las alteraciones del tono muscular y los movimientos involuntarios. Por ello, estos pacientes presentan dificultades en la coordinación y en el control de movimientos, lo cual les dificulta el caminar y afecta su postura cuando están sentados. Cabe resaltar que la capacidad cognitiva de las personas con DCP puede variar desde completamente normal, hasta un retraso mental severo. Las personas con DCP han sido seleccionadas como objetivo de esta tesis ya el margen de mejora de sus capacidades es amplio gracias a sistemas de control fisiológico, debido a sus múltiples y complejas limitaciones (problemas de habla y control motor). Debido a sus dificultades de comunicación, algunas personas con DCP se pueden comunicar con lenguaje escrito, siempre y cuando tenga un buen control motor. Sin embargo, la mayoría tienen dificultades para usar sistemas de Comunicación Aumentativos y Alternativos (AAC). De hecho, las personas con DCP utilizan generalmente tableros de conceptos para indicar la idea que quieren transmitir. Sin embargo, la mayoría las soluciones de comunicación disponibles en la actualidad están basadas en software propietario que hacen difícil la personalización del tablero de conceptos y el tipo de sistema de control. Es aquí donde surge esta tesis, con el objetivo de crear una interfaz con esas características, capaz de adaptarse a las necesidades y limitaciones del usuario. De esta forma, esta tesis propone un sistema de comunicación aumentativo y alternativo para personas con DCP basado en control fisiológico. Además, se propone un Sistema innovador de control directo sobre tableros de conceptos basado en EMG. Este Sistema de control se basa en un modelo físico que reproduce la respuesta mecánica muscular (basado en parámetros como Rigidez, Inercia y Viscosidad), permitiendo la selección de elementos gracias a pequeños pulsos de señal EMG con sensores sobre un músculo con control motor. Sus principales ventajas son la posibilidad de corregir errores durante la selección asociado a los impulsos musculares no controlados, evitar el esfuerzo muscular mantenido para alcanzar un nivel y reducir la fatiga.La Paràlisi Cerebral de tipus Discinèsica (DCP) es caracteritza principalment per les alteracions del to muscular i els moviments involuntaris. Per açò, aquests pacients presenten dificultats en la coordinació i en el control de moviments, la qual cosa els dificulta el caminar i afecta la seua postura quan estan asseguts. Cal ressaltar que la capacitat cognitiva de les persones amb DCP pot variar des de completament normal, fins a un retard mental sever. Les persones amb DCP han sigut seleccionades com a objectiu d'aquesta tesi ja el marge de millora de les seues capacitats és ampli gràcies a sistemes de control fisiològic, a causa dels seus múltiples i complexes limitacions (problemes de parla i control motor). A causa de les seues dificultats de comunicació, algunes persones amb DCP es poden comunicar amb llenguatge escrit, sempre que tinga un bon control motor. No obstant açò, la majoria tenen dificultats per a usar sistemes de Comunicació Augmentatius i Alternatius (AAC). De fet, les persones amb DCP utilitzen generalment taulers de conceptes per a indicar la idea que volen transmetre. No obstant açò, la majoria les solucions de comunicació disponibles en l'actualitat estan basades en programari propietari que fan difícil la personalització del tauler de conceptes i el tipus de sistema de control. És ací on sorgeix aquesta tesi, amb l'objectiu de crear una interfície amb aqueixes característiques, capaç d'adaptar-se a les necessitats i limitacions de l'usuari. D'aquesta forma, aquesta tesi proposa un sistema de comunicació augmentatiu i alternatiu per a persones amb DCP basat en control fisiològic. A més, es proposa un sistema innovador de control directe sobre taulers de conceptes basat en EMG. Aquest sistema de control es basa en un model físic que reprodueix la resposta mecànica muscular (basat en paràmetres com a Rigidesa, Inèrcia i Viscositat), permetent la selecció d'elements gràcies a xicotets polsos de senyal EMG amb sensors sobre un múscul amb control motor. Els seus principals avantatges són la possibilitat de corregir errors durant la selecció associat als impulsos musculars no controlats, evitar l'esforç muscular mantingut per a aconseguir un nivell i reduir la fatiga.Díaz Pineda, JA. (2017). Advanced Augmentative and Alternative Communication System Based in Physiological Control [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/90418TESI

Brain Computer Interfaces: Challenges to Clinical Viability Addressed in the Laboratory

Paralysis following spinal cord injuries, amyotrophic lateral sclerosis, stroke, and other disorders can intervene with signal transduction from the brain to the motor periphery, and eliminate the ability to perform volitional movements. Brain computer interfaces (BCI) directly measure brain activity associated with the user’s intent and translate the recorded brain activity into control signals for BCI applications, such as moving a computer cursor or a robot arm. While BCI technology has become an active and exciting field of research, much of the field’s development and achievements to date have taken place in the laboratory. The translation of BCI technology to the clinical setting is still not a reality. My thesis research has been dedicated to the objective of facilitating the translation of BCI systems from the primate lab to a clinical setting. That guiding objective has led me to work on several projects including: a technique that vastly improves the longevity of surgical implants in primate studies; a task that pushes the limits of sensorimotor performance – improving our knowledge of the function of primary motor cortex during realistic reaches and allowing us to quantify feedback effectiveness; characterizing the long-term tissue response to chronically implanted electrodes, and investigating how to optimally select parameters for neural information extraction. Each of these contributions will help bring BCI systems one step closer to clinical reality

The Internet of Things Will Thrive by 2025

This report is the latest research report in a sustained effort throughout 2014 by the Pew Research Center Internet Project to mark the 25th anniversary of the creation of the World Wide Web by Sir Tim Berners-LeeThis current report is an analysis of opinions about the likely expansion of the Internet of Things (sometimes called the Cloud of Things), a catchall phrase for the array of devices, appliances, vehicles, wearable material, and sensor-laden parts of the environment that connect to each other and feed data back and forth. It covers the over 1,600 responses that were offered specifically about our question about where the Internet of Things would stand by the year 2025. The report is the next in a series of eight Pew Research and Elon University analyses to be issued this year in which experts will share their expectations about the future of such things as privacy, cybersecurity, and net neutrality. It includes some of the best and most provocative of the predictions survey respondents made when specifically asked to share their views about the evolution of embedded and wearable computing and the Internet of Things

Using novel stimuli and alternative signal processing techniques to enhance BCI paradigms

A Brain-Computer Interface (BCI) is a device that uses the brain activity of a person as an input to select desired outputs on a computer. BCIs that use surface electroencephalogram (EEG) recordings as their input are the least invasive but also suffer from a very low signal-to-noise ratio (SNR) due to the very low amplitude of the person’s brain activity and the presence of many signal artefacts and background noise. This can be compensated for by subjecting the signals to extensive signal processing, and by using stimuli to trigger a large but consistent change in the signal – these changes are called evoked potentials. The method used to stimulate the evoked potential, and introduce an element of conscious selection in order to allow the user’s intent to modify the evoked potential produced, is called the BCI paradigm. However, even with these additions the performance of BCIs used for assistive communication and control is still significantly below that of other assistive solutions, such as keypads or eye-tracking devices. This thesis examines the paradigm and signal processing components of BCIs and puts forward several methods meant to enhance BCIs’ performance and efficiency. Firstly, two novel signal processing methods based on Empirical Mode Decomposition (EMD) were developed and evaluated. EMD is a technique that divides any oscillating signal into groups of frequency harmonics, called Intrinsic Mode Functions (IMFs). Furthermore, by using Takens’ theorem, a single channel of EEG can be converted into a multi-temporal channel signal by transforming the channel into multiple snapshots of its signal content in time using a series of delay vectors. This signal can then be decomposed into IMFs using a multi-channel variation of EMD, called Multi-variate EMD (MEMD), which uses the spatial information from the signal’s neighbouring channels to inform its decomposition. In the case of a multi-temporal channel signal, this allows the temporal dynamics of the signal to be incorporated into the IMFs. This is called Temporal MEMD (T-MEMD). The second signal processing method based on EMD decomposed both the spatial and temporal channels simultaneously, allowing both spatial and temporal dynamics to be incorporated into the resulting IMFs. This is called Spatio-temporal MEMD (ST-MEMD). Both methods were applied to a large pre-recorded Motor Imagery BCI dataset along with EMD and MEMD for comparison. These results were also compared to those from other studies in the literature that had used the same dataset. T-MEMD performed with an average classification accuracy of 70.2%, performing on a par with EMD that had an average classification accuracy of 68.9%. Both ST-MEMD and MEMD outperformed them with ST-MEMD having an average classification accuracy of 73.6%, and MEMD having an average classification accuracy of 75.3%. The methods containing spatial dynamics, i.e. MEMD and ST-MEMD, outperformed those with only temporal dynamics, i.e. EMD and T-MEMD. The two methods with temporal dynamics each performed on a par with the non-temporal method that had the same level of spatial dynamics. This shows that only the presence of spatial dynamics resulted in a performance increase. This was concluded to be because the differences between the classes of motor-imagery are inherently spatial in nature, not temporal. Next a novel BCI paradigm was developed based on the standard Steady-state Somatosensory Evoked Potential (SSSEP) BCI paradigm. This paradigm uses a tactile stimulus applied to the skin at a certain frequency, generating a resonance signal in the brain’s activity. If two stimuli of different frequency are applied, two resonance signals will be present. However, if the user attends one stimulus over the other, its corresponding SSSEP will increase in amplitude. Unfortunately these changes in amplitude can be very minute. To counter this, a stimulus amplitude and frequency of the vibrotactile stimuli. It was hypothesised that if the stimuli generator was constructed that could alter the were of the same frequency, but one’s amplitude was just below the user’s conscious level of perception and the other was above it, the changes in the SSSEP between classes would be the same as those between an SSSEP being generated and neutral EEG, with differences in α activity between the low-amplitude SSSEP and neutral activity due to the differences in the user’s level of concentration from attending the low-amplitude stimulus. The novel SSSEP BCI paradigm performed on a par with the standard paradigm with an average 61.8% classification accuracy over 16 participants, compared to an average 63.3% classification accuracy respectively, indicating that the hypothesis was false. However, the large presence of electro-magnetic interference (EMI) in the EEG recordings may have compromised the data. Many different noise suppression methods were applied to the stimulus device and the data, and whilst the EMI artefacts were reduced in magnitude they were not eliminated completely. Even with the noise the standard SSSEP stimulus paradigm performed on a par with studies that used the same paradigm, indicating that the results may not have been invalidated by the EMI. Overall the thesis shows that motor-imagery signals are inherently spatial in difference, and that the novel methods of T-MEMD and ST-MEMD may yet out-perform the existing methods of EMD and MEMD if applied to signals that are temporal in nature, such as functional Magnetic Resonance Imaging (fMRI). Whilst the novel SSSEP paradigm did not result in an increase in performance, it highlighted the impact of EMI from stimulus equipment on EEG recordings and potentially confirmed that the amplitude of SSEP stimuli is a minor factor in a BCI paradigm

Neurophysiological investigation of auditory augmentation to facilitate observational learning of everyday actions

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel UniversityThis thesis explores the neurophysiology of auditory augmentation of observed actions, and its effects on motor learning, neural activity, and plasticity. To this end, three studies were conducted. In the first study, we used Transcranial Magnetic Stimulation (TMS) to determine the effects of practising a motor task via sonification of combined action observation and motor imagery (sAOMI) on corticospinal excitability, compared to practising without extrinsic auditory information. In addition, we aimed at probing practice-dependent audiomotor plasticity. To this extent we used a variation of a commonly used method to probe and induce plasticity in humans, auditory paired associative stimulation (aPAS). Practice significantly increased corticospinal excitability, but sonification did not affect it. In addition, while aPAS completed alone significantly modulated corticospinal excitability, when practice primed aPAS, no neuromodulation was found. In a follow-up study, we explored the effects of sAOMI on corticospinal excitability during action observation (AO) or AOMI of the same action, and whether sonification induced audiomotor resonance, which is usually interpreted as development of an audiomotor association. The results corroborated and extended the findings of the previous study: practice increased corticospinal excitability at rest and during AO and AOMI, but sonification did not affect it. In addition, with sonification did not induce an audiomotor association. In a third study, we used electroencephalography (EEG) and other psychophysical measures, including a motor imagery questionnaire and mental chronometry, to assess changes resulting from practising with sonified action observation, followed by motor imagery of the same action. After two practice sessions, performance and kinaesthetic motor imagery vividness significantly improved, and participants’ mental chronometry was significantly more aligned with the speed of the observed action, compared to pre-practice measures. Sonification did not induce changes in any measure. EEG analysis revealed that participants who practised with sonification were able to sustain event-related desynchronization (ERD) in the lower alpha band (7-10 Hz) for longer, compared to participants who practices without sonification. No changes in higher alpha (10-12 Hz) or Beta (16-25 Hz) bands were found. Taken together, convergent results from this thesis suggest that sonified action observation has little effect on neurophysiological and behavioural markers of motor imagery ability and performance in healthy individuals. On the other hand, practising with sonified action observation may induce attentional modulations that enhance the learner’s ability to sustain action-related attentional processing for longer. We discuss these results in the context of contemporary neurocomputational theories of perception and action

Digital Interaction and Machine Intelligence

This book is open access, which means that you have free and unlimited access. This book presents the Proceedings of the 9th Machine Intelligence and Digital Interaction Conference. Significant progress in the development of artificial intelligence (AI) and its wider use in many interactive products are quickly transforming further areas of our life, which results in the emergence of various new social phenomena. Many countries have been making efforts to understand these phenomena and find answers on how to put the development of artificial intelligence on the right track to support the common good of people and societies. These attempts require interdisciplinary actions, covering not only science disciplines involved in the development of artificial intelligence and human-computer interaction but also close cooperation between researchers and practitioners. For this reason, the main goal of the MIDI conference held on 9-10.12.2021 as a virtual event is to integrate two, until recently, independent fields of research in computer science: broadly understood artificial intelligence and human-technology interaction