Effect of saccades in tongue electrotactile stimulation for vision substitution applications

The visual substitution paradigm aims to facilitate the life of blind people. Generally one uses electro-stimulating devices where electrodes are arranged into arrays to stimulate the skin or the tongue mucosa to send signals of visual type to the subjects. When an electro-stimulation signal is applied continuously (e.g. when static visual scenes are displayed for a long period of time), the receptors of the affected region can get saturated and the patient may lose the displayed information. We propose here some mechanisms that ameliorate the quality of perception of the electro-stimulation information. The electrical signal is encoded as 2D scenes projected onto the tongue via a Tongue Display Unit, i.e. an electro-tactile stimulator formed by a 12x12 matrix of electrodes. We propose to apply stochastic saccades on this signal. Our assumption is that this eye-inspired mechanism should make the visual substitution more efficient (by improving the perception) because of the reduction of the tactile receptors saturation. The influence of saccades was evaluated by a series of experiments. Results revealed a benefit on the persistence of perception due to saccades. This work helps to prevent the saturation of receptors on the tongue. Therefore increasing the quality of vision by the way of the electro-stimulation. It allows new enhancement features to retinal prosthesis devices which suffer from the same phenomenon

Electrotactile vision substitution for 3D trajectory following

Navigation for blind persons represents a challenge for researchers in vision substitution. In this field, one of the used techniques to navigate is guidance. In this study, we develop a new approach for 3D trajectory following in which the requested task is to track a light path using computer input devices (keyboard and mouse) or a rigid body handled in front of a stereoscopic camera. The light path is visualized either on direct vision or by way of a electro-stimulation device, the Tongue Display Unit, a 12x12 matrix of electrodes. We improve our method by a series of experiments in which the effect of the modality of perception and that of the input device. Preliminary results indicated a close correlation between the stimulated and recorded trajectories

Substitution visuelle par électro-stimulation linguale : étude des procédés affectant la qualité de perception

Our perception of the world is by essence multimodal. Various sensory cues (vision, hearing, touch, etc.) are hence integrated to allow us the execution of daily life activities. When one sense organ is missing or impaired, in some circumstances, the central nervous system can replace or reinforce the missing sensory information by using reliable sensory information coming from an other intact sense organ. This doctoral thesis was designed to explore this so-called ‘sensory substitution' concept through the exploitation of a visual-to-tactile sensory substitution device. the Tongue Display Unit (TDU) is made of a 12x12 electro-tactile matrix capable of rendering visual scences on the dorsal part of the tongue.However, the effectiveness of this device suffers from numerous drawbacks among which two main problems : (i) the problem of ‘fading' (mitigation) which corresponds to a rapid loss of perception of electrostimulation signal by the individuals and (ii) the problem of multimodal fusion. In the present work, we propose to investigate these problems and their effects on the quality of perception, by (i) applying some eye-inspired mechanisms (micro-saccades) to the electro-tactile signal on the tongue, (ii) studying how individuals perceive a picture signal with multi-modal perceptual (visual stimulus + substitution on the tongue) and (iii) the effect produced contradictions of information in this context.This work evidence the importance of applying saccades and sparkle mechanisms in electro-tactile signal to resolve the problem of fading especially when stimulation periods are relatively long. Furthermore, the experiments we have conducted on multi-modal perception and contradictions suggested that the perception of image information through the TDU was comparable to that of vision. Finally, we were able to identify a conflicting information fusion effect between visual perception and electrotactile tongue perception. This phenomenon is known as Mc Gurck effect (fusion of visual and auditory information). Understanding the effect of these processes to improve the quality of perception through the visual substitution paradigm and the importance of multi-modal fusion effects during this thesis could be applied in various areas where these effects are present or required, especially in the emerging field of prosthetic vision.Notre perception du monde est multimodale. Les différents sens (vision, audition, tactile, etc) sont coordonnés entre eux afin d'accomplir nos tâches quotidiennes. Dans le cas où un organe des sens connaît un dysfonctionnement, le système nerveux central est capable de remplacer ou de renforcer l'information manquante en faisant appel à d'autres modalités sensorielles intactes. Le cadre conceptuel de ce remplacement d'un sens par un autre est le paradigme de substitution sensorielle. Cette thèse se situe dans ce contexte et, pour explorer certains aspects de ce paradigme, exploite un dispositif d'électro-stimulation linguale, le Tongue Display Unit (TDU), une matrice de 12x12 électrodes destinée à restituer sous la forme de voltages variables des scènes visuelles sur la langue.L'efficacité de ce dispositif souffre de nombreux problèmes au rang desquels le problème de fading (atténuation) qui correspond à une perte rapide de perception du signal d'électro-stimulation par les sujets et celui de la fusion de perceptions multi-modales qui, nous le verrons, se complique avec l'usage du TDU. Dans cette thèse, nous proposons d'étudier l'effet sur la qualité de perception et sur l'atténuation du signal de l'adjonction au signal image de saccades et de tremblements, processus qui nous ont été inspirés par la vision. Nous étudions également la façon dont les sujets perçoivent un signal image en cas de multi-modalité perceptive (vision + substitution visuelle sur la langue) et l'effet que produisent des contradictions d'information dans ce contexte.Ce travail met en évidence l'importance de la présence de saccades ou de scintillement dans le signal d'électro-stimulation pour contrer les effets de fading : grâce à ces procédés, même lorsque les stimulations sont longues, les sujets continuent de percevoir correctement le signal image. De plus, les expériences de perception multi-modale et de contradictions nous ont permis de montrer que la perception d'informations de type image par l'intermédiaire du TDU était comparable à celle de la vision. Enfin, nous avons pu mettre en évidence un effet de fusion d'informations contradictoires entre la perception visuelle et linguale, connu dans le cadre de la fusion d'information visuo-auditives sous le nom d'effet Mc Gurck.La compréhension de l'effet de procédés destinés à améliorer la perception dans le cadre de la substitution visuelle et celle de l'importance des effets de fusion multi-modale dégagée au cours de thèse devraient trouver application dans de nombreux domaines où ces effets sont présents ou requis, en particulier dans le domaine émergent de la vision prothétique

Towards a better understanding of sensory substitution: the theory and practice of developing visual-to-auditory sensory substitution devices

Visual impairment is a global and potentially devastating affliction. Sensory substitution devices have the potential to lessen the impact of blindness by presenting vision via another modality. The chief motivation behind each of the chapters that follow is the production of more useful sensory substitution devices. The first empirical chapter (chapter two) demonstrates the use of interactive genetic algorithms to determine an optimal set of parameters for a sensory substitution device based on an auditory encoding of vision (“the vOICe”). In doing so, it introduces the first version of a novel sensory substitution device which is configurable at run-time. It also presents data from three interactive genetic algorithm based experiments that use this new sensory substitution device. Chapter three radically expands on this theme by introducing a general purpose, modular framework for developing visual-to-auditory sensory substitution devices (“Polyglot”). This framework is the fuller realisation of the Polyglot device introduced in the first chapter and is based on the principle of End-User Development (EUD). In chapter four, a novel method of evaluating sensory substitution devices using eye-tracking is introduced. The data shows both that the copresentation of visual stimuli assists localisation and that gaze predicted an auditory target location more reliably than the behavioural responses. Chapter five explores the relationship between sensory substitution devices and other tools that are used to acquire real-time sensory information (“sensory tools”). This taxonomy unites a range of technology from telescopes and cochlear implants to attempts to create a magnetic sense that can guide further research. Finally, in chapter six, the possibility of representing colour through sound is explored. The existence of a crossmodal correspondence between (equi-luminant) hue and pitch is documented that may reflect a relationship between pitch and the geometry of visible colour space

Designing sensory-substitution devices: Principles, pitfalls and potential1

An exciting possibility for compensating for loss of sensory function is to augment deficient senses by conveying missing information through an intact sense. Here we present an overview of techniques that have been developed for sensory substitution (SS) for the blind, through both touch and audition, with special emphasis on the importance of training for the use of such devices, while highlighting potential pitfalls in their design. One example of a pitfall is how conveying extra information about the environment risks sensory overload. Related to this, the limits of attentional capacity make it important to focus on key information and avoid redundancies. Also, differences in processing characteristics and bandwidth between sensory systems severely constrain the information that can be conveyed. Furthermore, perception is a continuous process and does not involve a snapshot of the environment. Design of sensory substitution devices therefore requires assessment of the nature of spatiotemporal continuity for the different senses. Basic psychophysical and neuroscientific research into representations of the environment and the most effective ways of conveying information should lead to better design of sensory substitution systems. Sensory substitution devices should emphasize usability, and should not interfere with other inter- or intramodal perceptual function. Devices should be task-focused since in many cases it may be impractical to convey too many aspects of the environment. Evidence for multisensory integration in the representation of the environment suggests that researchers should not limit themselves to a single modality in their design. Finally, we recommend active training on devices, especially since it allows for externalization, where proximal sensory stimulation is attributed to a distinct exterior object.This work was supported by the European Union’s Horizon 2020 Research and Innovative Programme under grant agreement no. 643636, “Sound of Vision”.Peer Reviewe

Somatic ABC's: A Theoretical Framework for Designing, Developing and Evaluating the Building Blocks of Touch-Based Information Delivery

abstract: Situations of sensory overload are steadily becoming more frequent as the ubiquity of technology approaches reality--particularly with the advent of socio-communicative smartphone applications, and pervasive, high speed wireless networks. Although the ease of accessing information has improved our communication effectiveness and efficiency, our visual and auditory modalities--those modalities that today's computerized devices and displays largely engage--have become overloaded, creating possibilities for distractions, delays and high cognitive load; which in turn can lead to a loss of situational awareness, increasing chances for life threatening situations such as texting while driving. Surprisingly, alternative modalities for information delivery have seen little exploration. Touch, in particular, is a promising candidate given that it is our largest sensory organ with impressive spatial and temporal acuity. Although some approaches have been proposed for touch-based information delivery, they are not without limitations including high learning curves, limited applicability and/or limited expression. This is largely due to the lack of a versatile, comprehensive design theory--specifically, a theory that addresses the design of touch-based building blocks for expandable, efficient, rich and robust touch languages that are easy to learn and use. Moreover, beyond design, there is a lack of implementation and evaluation theories for such languages. To overcome these limitations, a unified, theoretical framework, inspired by natural, spoken language, is proposed called Somatic ABC's for Articulating (designing), Building (developing) and Confirming (evaluating) touch-based languages. To evaluate the usefulness of Somatic ABC's, its design, implementation and evaluation theories were applied to create communication languages for two very unique application areas: audio described movies and motor learning. These applications were chosen as they presented opportunities for complementing communication by offloading information, typically conveyed visually and/or aurally, to the skin. For both studies, it was found that Somatic ABC's aided the design, development and evaluation of rich somatic languages with distinct and natural communication units.Dissertation/ThesisPh.D. Computer Science 201

Perturbation-based detection and prosthetic correction of vestibulopathic gait

Thesis (Ph. D.)--Harvard-MIT Division of Health Sciences and Technology, February 2007.Includes bibliographical references.While being able to balance is something most of us take for granted, each year approximately 400,000 Americans are diagnosed with a balance disorder. In order to prevent fall-related injuries due to postural instability, it is important to create both diagnosis techniques so that therapy can be applied before a fall occurs and devices which can aid the balance-impaired population. The aims of this research are twofold: 1) to develop metrics that quantify the locomotor stability of individuals with reduced vestibular function and 2) to assess the capability of a noninvasive vibrotactile balance prosthesis for improving postural and gait stability. The clinical standards of practice for assessing vestibular deficiency include testing postural stability while standing but not during locomotion. This research examines one prospective locomotor-based technique involving the analysis of postural recovery from controlled surface perturbations. The research also investigates the use of a novel wearable vibrotactile sensory substitution device for enhanced postural and locomotor stability. The balance prosthesis is composed of an inertial motion-sensing system mounted on the lower back, a vibrotactile display worn around the torso, and a computer controller.(cont.) It can serve as a permanent or temporary replacement of motion cues, a tool for vestibular rehabilitation, or an additional sensory channel for military troops, pilots, and astronauts. This research demonstrates that well-compensated vestibulopathic patients can be differentiated from young and age-matched controls during over ground locomotion based on step width variability. Prior to this research, unilateral and bilateral vestibulopathic patients donning the vibrotactile balance prosthesis have demonstrated increased postural stability during single-axis support surface perturbations using single-axis sway information. This work shows that multi-directional vibrotactile tilt feedback reduces postural sway during multi-directional support surface perturbations, and has both short- and long-term effects on increasing postural stability. Finally, this research demonstrates for the first time that medial-lateral (M/L) tilt feedback can be used by balance-deficient subjects to reduce factors associated with fall risk (M/L tilt and M/L step width variability) during various locomotor tasks.by Kathleen H. Sienko.Ph.D

Vision artificielle pour les non-voyants : une approche bio-inspirée pour la reconnaissance de formes

More than 315 million people worldwide suffer from visual impairments, with several studies suggesting that this number will double by 2030 due to the ageing of the population. To compensate for the loss of sight the current approaches consist of either specific aids designed to answer particular needs or generic systems such as neuroprostheses and sensory substitution devices. These holistic approaches, which try to restore vision as a whole, have been shown to be very inefficient in real life situations given the low resolution of output interfaces. To overcome these obstacles we propose the use of artificial vision in order to pre-process visual scenes and provide the user with relevant information. We have validated this approach through the development of a novel assistive device for the blind called Navig. Through shape recognition and spatialized sounds synthesis, this system allows users to locate and grab objects of interest. It also features navigational aids based on a new positioning method combining GPS, inertial sensors and the visual detection of geolocalized landmarks. To enhance the performance of the visual module we further developed, as part of this thesis, a bio-inspired pattern recognition algorithm which uses latency-based coding of visual information, oriented edge representations and a cascaded architecture combining detection at different resolutions.La déficience visuelle touche aujourd’hui plus de 315 millions de personnes à travers le monde, un chiffre qui pourrait doubler d’ici à 2030 du fait du vieillissement de la population. Les deux grandes approches existantes pour compenser la perte de vision sont les aides spécifiques, répondant à un besoin identifié, et les systèmes génériques tels que les neuroprothèses ou les systèmes de substitution sensorielle. Ces approches holistiques, tentant de restituer l’ensemble de l’information visuelle, s’avèrent inadaptées de par la trop faible résolution des interfaces de sortie, rendant ces systèmes inutilisables dans la vie quotidienne. Face à ce constat, nous proposons dans cette thèse une démarche alternative, consistant à intégrer des méthodes de vision artificielle, afin de prétraiter la scène visuelle, et de ne restituer au non-voyant que les informations extraites pertinentes. Pour valider cette approche, nous présenterons le développement d’un système de suppléance baptisé Navig. Grâce à la reconnaissance de formes et à la synthèse de sons spatialisés, il permet à l’utilisateur de localiser des objets d’intérêt. Il offre également des fonctions de navigation, basées sur une nouvelle méthode de positionnement combinant GPS, données inertielles, et détections de cibles visuelles géolocalisées. Afin d’améliorer les performances du module de vision artificielle, nous proposerons également dans cette thèse un nouvel algorithme de reconnaissance de formes bio-inspiré, reposant sur un codage de l’information visuelle par latence, sur des représentations sous forme d’arêtes orientées, et sur une architecture en cascade combinant des détections à différentes résolutions