Touch- and Walkable Virtual Reality to Support Blind and Visually Impaired Peoples‘ Building Exploration in the Context of Orientation and Mobility

Der Zugang zu digitalen Inhalten und Informationen wird immer wichtiger für eine erfolgreiche Teilnahme an der heutigen, zunehmend digitalisierten Zivilgesellschaft. Solche Informationen werden meist visuell präsentiert, was den Zugang für blinde und sehbehinderte Menschen einschränkt. Die grundlegendste Barriere ist oft die elementare Orientierung und Mobilität (und folglich die soziale Mobilität), einschließlich der Erlangung von Kenntnissen über unbekannte Gebäude vor deren Besuch. Um solche Barrieren zu überbrücken, sollten technische Hilfsmittel entwickelt und eingesetzt werden. Es ist ein Kompromiss zwischen technologisch niedrigschwellig zugänglichen und verbreitbaren Hilfsmitteln und interaktiv-adaptiven, aber komplexen Systemen erforderlich. Die Anpassung der Technologie der virtuellen Realität (VR) umfasst ein breites Spektrum an Entwicklungs- und Entscheidungsoptionen. Die Hauptvorteile der VR-Technologie sind die erhöhte Interaktivität, die Aktualisierbarkeit und die Möglichkeit, virtuelle Räume und Modelle als Abbilder von realen Räumen zu erkunden, ohne dass reale Gefahren und die begrenzte Verfügbarkeit von sehenden Helfern auftreten. Virtuelle Objekte und Umgebungen haben jedoch keine physische Beschaffenheit. Ziel dieser Arbeit ist es daher zu erforschen, welche VR-Interaktionsformen sinnvoll sind (d.h. ein angemessenes Verbreitungspotenzial bieten), um virtuelle Repräsentationen realer Gebäude im Kontext von Orientierung und Mobilität berührbar oder begehbar zu machen. Obwohl es bereits inhaltlich und technisch disjunkte Entwicklungen und Evaluationen zur VR-Technologie gibt, fehlt es an empirischer Evidenz. Zusätzlich bietet diese Arbeit einen Überblick über die verschiedenen Interaktionen. Nach einer Betrachtung der menschlichen Physiologie, Hilfsmittel (z.B. taktile Karten) und technologischen Eigenschaften wird der aktuelle Stand der Technik von VR vorgestellt und die Anwendung für blinde und sehbehinderte Nutzer und der Weg dorthin durch die Einführung einer neuartigen Taxonomie diskutiert. Neben der Interaktion selbst werden Merkmale des Nutzers und des Geräts, der Anwendungskontext oder die nutzerzentrierte Entwicklung bzw. Evaluation als Klassifikatoren herangezogen. Begründet und motiviert werden die folgenden Kapitel durch explorative Ansätze, d.h. im Bereich 'small scale' (mit sogenannten Datenhandschuhen) und im Bereich 'large scale' (mit einer avatargesteuerten VR-Fortbewegung). Die folgenden Kapitel führen empirische Studien mit blinden und sehbehinderten Nutzern durch und geben einen formativen Einblick, wie virtuelle Objekte in Reichweite der Hände mit haptischem Feedback erfasst werden können und wie verschiedene Arten der VR-Fortbewegung zur Erkundung virtueller Umgebungen eingesetzt werden können. Daraus werden geräteunabhängige technologische Möglichkeiten und auch Herausforderungen für weitere Verbesserungen abgeleitet. Auf der Grundlage dieser Erkenntnisse kann sich die weitere Forschung auf Aspekte wie die spezifische Gestaltung interaktiver Elemente, zeitlich und räumlich kollaborative Anwendungsszenarien und die Evaluation eines gesamten Anwendungsworkflows (d.h. Scannen der realen Umgebung und virtuelle Erkundung zu Trainingszwecken sowie die Gestaltung der gesamten Anwendung in einer langfristig barrierefreien Weise) konzentrieren.Access to digital content and information is becoming increasingly important for successful participation in today's increasingly digitized civil society. Such information is mostly presented visually, which restricts access for blind and visually impaired people. The most fundamental barrier is often basic orientation and mobility (and consequently, social mobility), including gaining knowledge about unknown buildings before visiting them. To bridge such barriers, technological aids should be developed and deployed. A trade-off is needed between technologically low-threshold accessible and disseminable aids and interactive-adaptive but complex systems. The adaptation of virtual reality (VR) technology spans a wide range of development and decision options. The main benefits of VR technology are increased interactivity, updatability, and the possibility to explore virtual spaces as proxies of real ones without real-world hazards and the limited availability of sighted assistants. However, virtual objects and environments have no physicality. Therefore, this thesis aims to research which VR interaction forms are reasonable (i.e., offering a reasonable dissemination potential) to make virtual representations of real buildings touchable or walkable in the context of orientation and mobility. Although there are already content and technology disjunctive developments and evaluations on VR technology, there is a lack of empirical evidence. Additionally, this thesis provides a survey between different interactions. Having considered the human physiology, assistive media (e.g., tactile maps), and technological characteristics, the current state of the art of VR is introduced, and the application for blind and visually impaired users and the way to get there is discussed by introducing a novel taxonomy. In addition to the interaction itself, characteristics of the user and the device, the application context, or the user-centered development respectively evaluation are used as classifiers. Thus, the following chapters are justified and motivated by explorative approaches, i.e., in the group of 'small scale' (using so-called data gloves) and in the scale of 'large scale' (using an avatar-controlled VR locomotion) approaches. The following chapters conduct empirical studies with blind and visually impaired users and give formative insight into how virtual objects within hands' reach can be grasped using haptic feedback and how different kinds of VR locomotion implementation can be applied to explore virtual environments. Thus, device-independent technological possibilities and also challenges for further improvements are derived. On the basis of this knowledge, subsequent research can be focused on aspects such as the specific design of interactive elements, temporally and spatially collaborative application scenarios, and the evaluation of an entire application workflow (i.e., scanning the real environment and exploring it virtually for training purposes, as well as designing the entire application in a long-term accessible manner)

Perception of Time-Discrete Haptic Feedback on the Waist is Invariant with Gait Events

The effectiveness of haptic feedback devices highly depends on the perception of tactile stimuli, which differs across body parts and can be affected by movement. In this study, a novel wearable sensory feedback apparatus made of a pair of pressure-sensitive insoles and a belt equipped with vibrotactile units is presented; the device provides time-discrete vibrations around the waist, synchronized with biomechanically-relevant gait events during walking. Experiments with fifteen healthy volunteers were carried out to investigate users' tactile perception on the waist. Stimuli of different intensities were provided at twelve locations, each time synchronously with one pre-defined gait event (i.e. heel strike, flat foot or toe off), following a pseudo-random stimulation sequence. Reaction time, detection rate and localization accuracy were analyzed as functions of the stimulation level and site and the effect of gait events on perception was investigated. Results revealed that above-threshold stimuli (i.e. vibrations characterized by acceleration amplitudes of 1.92g and 2.13g and frequencies of 100 Hz and 150 Hz, respectively) can be effectively perceived in all the sites and successfully localized when the intertactor spacing is set to 10 cm. Moreover, it was found that perception of time-discrete vibrations was not affected by phase-related gating mechanisms, suggesting that the waist could be considered as a preferred body region for delivering haptic feedback during walking

Portable haptic device for lower limb amputee gait feedback: Assessing static and dynamic perceptibility

Loss of joints and severed sensory pathway cause reduced mobility capabilities in lower limb amputees. Although prosthetic devices attempt to restore normal mobility functions, lack of awareness and control of limb placement increase the risk of falling and causing amputee to have high level of visual dependency. Haptic feedback can serve as a cue for gait events during ambulation thus providing sense of awareness of the limb position. This paper presents a wireless wearable skin stretch haptic device to be fitted around the thigh region. The movement profile of the device was characterized and a preliminary work with able-bodied participants and an above-knee amputee to assess the ability of users to perceive the delivered stimuli during static and dynamic mode is reported. Perceptibility was found to be increasing with stretch magnitude. It was observed that a higher magnitude of stretch was needed for the stimuli to be accurately perceived during walking in comparison to static standing, most likely due to the intense movement of the muscle and increased motor skills demand during walking activity

The Promise of Stochastic Resonance in Falls Prevention

Multisensory integration is essential for maintenance of motor and cognitive abilities, thereby ensuring normal function and personal autonomy. Balance control is challenged during senescence or in motor disorders, leading to potential falls. Increased uncertainty in sensory signals is caused by a number of factors including noise, defined as a random and persistent disturbance that reduces the clarity of information. Counter-intuitively, noise can be beneficial in some conditions. Stochastic resonance is a mechanism whereby a particular level of noise actually enhances the response of non-linear systems to weak sensory signals. Here we review the effects of stochastic resonance on sensory modalities and systems directly involved in balance control. We highlight its potential for improving sensorimotor performance as well as cognitive and autonomic functions. These promising results demonstrate that stochastic resonance represents a flexible and non-invasive technique that can be applied to different modalities simultaneously. Finally we point out its benefits for a variety of scenarios including in ambulant elderly, skilled movements, sports and to patients with sensorimotor or autonomic dysfunctions.Multisensory integration is essential for maintenance of motor and cognitive abilities, thereby ensuring normal function and personal autonomy. Balance control is challenged during senescence or in motor disorders, leading to potential falls. Increased uncertainty in sensory signals is caused by a number of factors including noise, defined as a random and persistent disturbance that reduces the clarity of information. Counter-intuitively, noise can be beneficial in some conditions. Stochastic resonance is a mechanism whereby a particular level of noise actually enhances the response of non-linear systems to weak sensory signals. Here we review the effects of stochastic resonance on sensory modalities and systems directly involved in balance control. We highlight its potential for improving sensorimotor performance as well as cognitive and autonomic functions. These promising results demonstrate that stochastic resonance represents a flexible and non-invasive technique that can be applied to different modalities simultaneously. Finally we point out its benefits for a variety of scenarios including in ambulant elderly, skilled movements, sports and to patients with sensorimotor or autonomic dysfunctions

Viia-hand: a Reach-and-grasp Restoration System Integrating Voice interaction, Computer vision and Auditory feedback for Blind Amputees

Visual feedback plays a crucial role in the process of amputation patients completing grasping in the field of prosthesis control. However, for blind and visually impaired (BVI) amputees, the loss of both visual and grasping abilities makes the "easy" reach-and-grasp task a feasible challenge. In this paper, we propose a novel multi-sensory prosthesis system helping BVI amputees with sensing, navigation and grasp operations. It combines modules of voice interaction, environmental perception, grasp guidance, collaborative control, and auditory/tactile feedback. In particular, the voice interaction module receives user instructions and invokes other functional modules according to the instructions. The environmental perception and grasp guidance module obtains environmental information through computer vision, and feedbacks the information to the user through auditory feedback modules (voice prompts and spatial sound sources) and tactile feedback modules (vibration stimulation). The prosthesis collaborative control module obtains the context information of the grasp guidance process and completes the collaborative control of grasp gestures and wrist angles of prosthesis in conjunction with the user's control intention in order to achieve stable grasp of various objects. This paper details a prototyping design (named viia-hand) and presents its preliminary experimental verification on healthy subjects completing specific reach-and-grasp tasks. Our results showed that, with the help of our new design, the subjects were able to achieve a precise reach and reliable grasp of the target objects in a relatively cluttered environment. Additionally, the system is extremely user-friendly, as users can quickly adapt to it with minimal training

Design of a vibrotactile stimulus paradigm for a biofeedback device to improve gait rehabilitation of lower limb amputees

Dissertação de mestrado integrado em Biomedical Engineering (specialization in Biomaterials Rehabilitation and Biomechanics)A lower limb amputation not only affects locomotion, but also the amputee's somatosensory system, body perception, and mental health and, naturally, the fear of falling is more pronounced. Consequently, the patient is faced with the challenge of developing motor strategies that allow him to carry out daily activities since the use of the prosthesis does not fully compensate for the deficiencies acquired by a prosthetic gait, such as, for instance, asymmetry and variation in the duration of the gait events. Faced with the absence of effective treatments that restore locomotor functionality, the BioWalk project presents a rehabilitation solution: a biofeedback system that assists amputees during gait training sessions. This system consists in applying a vibrotactile stimulus on the skin of the affected leg. This stimulus can be activated at different moments of the prosthetic gait, allowing the patient to have a better perception and awareness of his body and locomotion to be able to detect any abnormal motor behaviours during the rehabilitation sessions and, in the future, to establish an adequate and healthy gait pattern. Consequently, there is a need to analyse muscular and kinematic data of the gait of amputees to detect which events are critical in prosthetic gait, which muscles are activated or most required in gait, how the centre of mass behaves in the gait of an amputee, among other parameters. Thus, in this dissertation, the main goal is to investigate and propose the best way (i.e., paradigm) to apply a vibrotactile stimulus to be used in a biofeedback device during rehabilitation sessions.Uma amputação do membro inferior não afeta apenas a locomoção, mas também o sistema somatosensorial do amputado, a sua perceção corporal, a sua saúde mental e, naturalmente, o medo de cair encontra-se mais acentuado. Consequentemente, o paciente é confrontado com o desafio de desenvolver estratégias motoras que lhe permitam a realização de atividades diárias dado que o uso da prótese não compensa totalmente as deficiências adquiridas por uma marcha protética, como por exemplo, a assimetria e a variação na duração dos eventos de marcha. Perante a ausência de tratamentos eficazes que restaurem a funcionalidade locomotora, o projeto BioWalk apresenta uma solução de reabilitação: um sistema de biofeedback que auxilie a pessoa amputada durante sessões de treino de marcha. Este sistema consiste na aplicação de um estímulo vibrotátil sobre a pele da perna afetada. Este estímulo pode ser ativado em diversos momentos da marcha protética permitindo ao paciente uma melhor percetibilidade e consciência sobre o seu corpo e locomoção para que seja capaz de detetar algum comportamento motor anormal durante as sessões de reabilitação e para, futuramente, estabelecer um padrão de marcha adequado e saudável. Consequentemente, surge a necessidade de analisar dados musculares e cinemáticos da marcha de amputados de forma a detetar quais os eventos críticos na marcha protética, quais são os músculos ativados ou os que são mais requeridos na marcha, como se comporta o centro de massa na marcha de um amputado, entre outros parâmetros. Assim, nesta dissertação, o objetivo é propor um paradigma de estímulos vibrotáteis para serem usados num dispositivo de biofeedback durante sessões de reabilitação