A Human-Centric Metaverse Enabled by Brain-Computer Interface: A Survey

The growing interest in the Metaverse has generated momentum for members of academia and industry to innovate toward realizing the Metaverse world. The Metaverse is a unique, continuous, and shared virtual world where humans embody a digital form within an online platform. Through a digital avatar, Metaverse users should have a perceptual presence within the environment and can interact and control the virtual world around them. Thus, a human-centric design is a crucial element of the Metaverse. The human users are not only the central entity but also the source of multi-sensory data that can be used to enrich the Metaverse ecosystem. In this survey, we study the potential applications of Brain-Computer Interface (BCI) technologies that can enhance the experience of Metaverse users. By directly communicating with the human brain, the most complex organ in the human body, BCI technologies hold the potential for the most intuitive human-machine system operating at the speed of thought. BCI technologies can enable various innovative applications for the Metaverse through this neural pathway, such as user cognitive state monitoring, digital avatar control, virtual interactions, and imagined speech communications. This survey first outlines the fundamental background of the Metaverse and BCI technologies. We then discuss the current challenges of the Metaverse that can potentially be addressed by BCI, such as motion sickness when users experience virtual environments or the negative emotional states of users in immersive virtual applications. After that, we propose and discuss a new research direction called Human Digital Twin, in which digital twins can create an intelligent and interactable avatar from the user's brain signals. We also present the challenges and potential solutions in synchronizing and communicating between virtual and physical entities in the Metaverse

Ubiquitous Computing in a Home Environment, Controlling Consumer Electronics

Building interaction prototypes for ubiquitous computing is inherently difficult, since it involves a number of different devices and systems. Prototyping is an important step in developing and evaluating interaction concepts. The ideal prototyping methodology should offer high fidelity at a relatively low cost. This thesis describes the development of interaction concepts for controlling consumer electronics in a ubiquitous computing home environment, as well as the setup, based on immersive virtual reality, used to develop and evaluate the interaction concepts. Off-the-shelf input/output devices and a game engine are used for developing two concepts for device discovery and two concepts for device interaction. The interaction concepts are compared in a controlled experiment in order to evaluate the concepts as well as the virtual reality setup. Statistically significant differences and subjective preferences could be observed in the quantitative and qualitative data respectively. Overall, the results suggest that the interaction concepts could be acceptable to some users for some use cases and that the virtual reality setup offers the possibility to quickly build interaction concepts which can be evaluated and compared in a controlled experiment

Around-Body Interaction: Leveraging Limb Movements for Interacting in a Digitally Augmented Physical World

Recent technological advances have made head-mounted displays (HMDs) smaller and untethered, fostering the vision of ubiquitous interaction with information in a digitally augmented physical world. For interacting with such devices, three main types of input - besides not very intuitive finger gestures - have emerged so far: 1) Touch input on the frame of the devices or 2) on accessories (controller) as well as 3) voice input. While these techniques have both advantages and disadvantages depending on the current situation of the user, they largely ignore the skills and dexterity that we show when interacting with the real world: Throughout our lives, we have trained extensively to use our limbs to interact with and manipulate the physical world around us. This thesis explores how the skills and dexterity of our upper and lower limbs, acquired and trained in interacting with the real world, can be transferred to the interaction with HMDs. Thus, this thesis develops the vision of around-body interaction, in which we use the space around our body, defined by the reach of our limbs, for fast, accurate, and enjoyable interaction with such devices. This work contributes four interaction techniques, two for the upper limbs and two for the lower limbs: The first contribution shows how the proximity between our head and hand can be used to interact with HMDs. The second contribution extends the interaction with the upper limbs to multiple users and illustrates how the registration of augmented information in the real world can support cooperative use cases. The third contribution shifts the focus to the lower limbs and discusses how foot taps can be leveraged as an input modality for HMDs. The fourth contribution presents how lateral shifts of the walking path can be exploited for mobile and hands-free interaction with HMDs while walking.Comment: thesi

Inclusive Augmented and Virtual Reality: A Research Agenda

Augmented and virtual reality experiences present significant barriers for disabled people, making it challenging to fully engage with immersive platforms. Whilst researchers have started to explore potential solutions addressing these accessibility issues, we currently lack a comprehensive understanding of research areas requiring further investigation to support the development of inclusive AR/VR systems. To address current gaps in knowledge, we led a series of multidisciplinary sandpits with relevant stakeholders (i.e., academic researchers, industry specialists, people with lived experience of disability, assistive technologists, and representatives from disability organisations, charities, and special needs educational institutions) to collaboratively explore research challenges, opportunities, and solutions. Based on insights shared by participants, we present a research agenda identifying key areas where further work is required in relation to specific forms of disability (i.e., across the spectrum of physical, visual, cognitive, and hearing impairments), including wider considerations associated with the development of more accessible immersive platforms

Exploration of digital biomarkers in chronic low back pain and Parkinson’s disease

Chronic pain and Parkinson’s disease are illnesses with personal disease progression, symptoms, and the experience of these. The ability to measure and monitor the symptoms by digitally and remotely is still limited. The aim was to study the usability and feasibility of real-world data from wearables, mobile devices, and patients in exploring digital biomarkers in these diseases. The key hypothesis was that this allows us to measure, analyse and detect clinically valid digital signals in movement, heart rate and skin conductance data. The laboratory grade data in chronic pain were collected in an open feasibility study by using a program and built-in sensors in virtual reality devices. The real-world data were collected with a randomized clinical study by clinical assessments, built-in sensors, and two wearables. The laboratory grade dataset in Parkinson’s disease was obtained from Michael J. Fox Foundation. It contained sensor data from three wearables with clinical assessments. The real-world data were collected with a clinical study by clinical assessments, a wearable, and a mobile application. With both diseases the laboratory grade data were first explored, before the real-world data were analyzed. The classification of chronic pain patients with the laboratory grade movement data was possible with a high accuracy. A novel real-world digital signal that correlates with clinical outcomes was found in chronic low back pain patients. A model that was able to detect different movement states was developed with laboratory grade Parkinson’s disease data. A detection of these states followed by the quantification of symptoms was found to be a potential method for the future. The usability of data collection methods in both diseases were found promising. In the future the analyses of movement data in these diseases could be further researched and validated as a movement based digital biomarkers to be used as a surrogate or additional endpoint. Combining the data science with the optimal usability enables the exploitation of digital biomarkers in clinical trials and treatment.Digitaalisten biomarkkereiden tunnistaminen kroonisessä alaselkäkivussa ja Parkinsonin taudissa Krooninen kipu ja Parkinsonin tauti ovat oireiden, oirekokemuksen sekä taudin kehittymisen osalta yksilöllisiä sairauksia. Kyky mitata ja seurata oireita etänä on vielä alkeellista. Väitöskirjassa tutkittiin kaupallisten mobiili- ja älylaitteiden hyödyntämistä digitaalisten biomarkkereiden löytämisessä näissä taudeissa. Pääolettamus oli, että kaupallisten älylaitteiden avulla kyetään tunnistamaan kliinisesti hyödyllisiä digitaalisia signaaleja. Kroonisen kivun laboratorio-tasoinen data kerättiin tätä varten kehitettyä ohjelmistoa sekä kaupallisia antureita käyttäen. Reaaliaikainen kipudata kerättiin erillisen hoito-ohjelmiston tehoa ja turvallisuutta mitanneessa kliinisessä tutkimuksessa sekä kliinisiä arviointeja että anturidataa hyödyntäen. Laboratorio-tasoinena datana Parkinsonin taudissa käytettiin Michael J. Fox Foundationin kolmella eri älylaitteella ja kliinisin arvioinnein kerättyä dataa. Reaaliaikainen data kerättiin käyttäen kliinisia arviointeja, älyranneketta ja mobiilisovellusta. Molempien indikaatioiden kohdalla laboratoriodatalle tehtyä eksploratiivista analyysia hyödynnettiin itse reaaliaikaisen datan analysoinnissa. Kipupotilaiden tunnistaminen laboratorio-tasoisesta liikedatasta oli mahdollista korkealla tarkkuudella. Reaaliaikaisesta liikedatasta löytyi uusi kliinisten arviointien kanssa korreloiva digitaalinen signaali. Parkinsonin taudin datasta kehitettiin uusi liiketyyppien tunnistamiseen tarkoitettu koneoppimis-malli. Sen hyödyntäminen liikedatan liiketyyppien tunnistamisessa ennen varsinaista oireiden mittausta on lupaava menetelmä. Käytettävyys molempien tautien reaaliaikaisissa mittausmenetelmissä havaittiin toimivaksi. Reaaliaikaiseen, kaupallisin laittein kerättävään liikedataan pohjautuvat digitaaliset biomarkkerit ovat lupaava kohde jatkotutkimukselle. Uusien analyysimenetelmien yhdistäminen optimaaliseen käytettävyyteen mahdollistaa tulevaisuudessa digitaalisten biomarkkereiden hyödyntämisen sekä kroonisten tautien kliinisessä tutkimuksessa että itse hoidossa