The evolution of a visual-to-auditory sensory substitution device using interactive genetic algorithms

Sensory Substitution is a promising technique for mitigating the loss of a sensory modality. Sensory Substitution Devices (SSDs) work by converting information from the impaired sense (e.g. vision) into another, intact sense (e.g. audition). However, there are a potentially infinite number of ways of converting images into sounds and it is important that the conversion takes into account the limits of human perception and other user-related factors (e.g. whether the sounds are pleasant to listen to). The device explored here is termed “polyglot” because it generates a very large set of solutions. Specifically, we adapt a procedure that has been in widespread use in the design of technology but has rarely been used as a tool to explore perception – namely Interactive Genetic Algorithms. In this procedure, a very large range of potential sensory substitution devices can be explored by creating a set of ‘genes’ with different allelic variants (e.g. different ways of translating luminance into loudness). The most successful devices are then ‘bred’ together and we statistically explore the characteristics of the selected-for traits after multiple generations. The aim of the present study is to produce design guidelines for a better SSD. In three experiments we vary the way that the fitness of the device is computed: by asking the user to rate the auditory aesthetics of different devices (Experiment 1), by measuring the ability of participants to match sounds to images (Experiment 2) and the ability to perceptually discriminate between two sounds derived from similar images (Experiment 3). In each case the traits selected for by the genetic algorithm represent the ideal SSD for that task. Taken together, these traits can guide the design of a better SSD

Technology inspired design for pervasive healthcare

Pervasive healthcare technologies are increasingly using novel sensory devices that are able to measure phenomena that could not be measured before. To develop novel healthcare applications that use these largely untested technologies, it is important to have a design process that allows proper exploration of the capabilities of the novel technologies. We focus on the technology-inspired design process that was used in the development of a system to support posture and provide guidance by nudging people, and how this has lead us to explore pervasive healthcare applications

Toward a Phenomenological Pragmatics of Enactive Perception

The enactive approach to perception is generating an extensive amount of interest and debate in the cognitive sciences. One particularly contentious issue has been how best to characterize the perceptual experiences reported by subjects who have mastered the skillful use of a perceptual supplementation (PS) device. This paper argues that this issue cannot be resolved with the use of third-person methodologies alone, but that it requires the development of a phenomenological pragmatics. In particular, it is necessary that the experimenters become skillful in the use of PS devices themselves. The "Enactive Torch" is proposed as an experimental platform which is cheap, non-intrusive and easy to replicate, so as to enable researchers to corroborate reported experiences with their own phenomenology more easily

In the dark – designing navigation for a haptic theatre experience

We describe the ongoing work of a collaborative theatre project, in which blind and sighted members of an interdisciplinary team are working on the creation of an immersive and accessible theatre experience. The theatre experience is set in pitch dark, and uses audio and haptic ubiquitous technologies to guide both blind and sighted members of the audience. During two sessions which explored the use of enactive navigation in the dark space, participants were able to locate areas of interest within the dark, using enactive explorative navigation, and some participants described this as a ‘dense’ experience

Low-fi skin vision: A case study in rapid prototyping a sensory substitution system

We describe the design process we have used to develop a minimal, twenty vibration motor Tactile Vision Sensory Substitution (TVSS) system which enables blind-folded subjects to successfully track and bat a rolling ball and thereby experience 'skin vision'. We have employed a low-fi rapid prototyping approach to build this system and argue that this methodology is particularly effective for building embedded interactive systems. We support this argument in two ways. First, by drawing on theoretical insights from robotics, a discipline that also has to deal with the challenge of building complex embedded systems that interact with their environments; second, by using the development of our TVSS as a case study: describing the series of prototypes that led to our successful design and highlighting what we learnt at each stage

The structure of experience, the nature of the visual, and type 2 blindsight

Unlike those with type 1 blindsight, people who have type 2 blindsight have some sort of consciousness of the stimuli in their blind field. What is the nature of that consciousness? Is it visual experience? I address these questions by considering whether we can establish the existence of any structural—necessary—features of visual experience. I argue that it is very difficult to establish the existence of any such features. In particular, I investigate whether it is possible to visually, or more generally perceptually, experience form or movement at a distance from our body, without experiencing colour. The traditional answer, advocated by Aristotle, and some other philosophers, up to and including the present day, is that it is not and hence colour is a structural feature of visual experience. I argue that there is no good reason to think that this is impossible, and provide evidence from four cases—sensory substitution, achomatopsia, phantom contours and amodal completion—in favour of the idea that it is possible. If it is possible then one important reason for rejecting the idea that people with type 2 blindsight do not have visual experiences is undermined. I suggest further experiments that could be done to help settle the matter

A robot hand testbed designed for enhancing embodiment and functional neurorehabilitation of body schema in subjects with upper limb impairment or loss.

Many upper limb amputees experience an incessant, post-amputation "phantom limb pain" and report that their missing limbs feel paralyzed in an uncomfortable posture. One hypothesis is that efferent commands no longer generate expected afferent signals, such as proprioceptive feedback from changes in limb configuration, and that the mismatch of motor commands and visual feedback is interpreted as pain. Non-invasive therapeutic techniques for treating phantom limb pain, such as mirror visual feedback (MVF), rely on visualizations of postural changes. Advances in neural interfaces for artificial sensory feedback now make it possible to combine MVF with a high-tech "rubber hand" illusion, in which subjects develop a sense of embodiment with a fake hand when subjected to congruent visual and somatosensory feedback. We discuss clinical benefits that could arise from the confluence of known concepts such as MVF and the rubber hand illusion, and new technologies such as neural interfaces for sensory feedback and highly sensorized robot hand testbeds, such as the "BairClaw" presented here. Our multi-articulating, anthropomorphic robot testbed can be used to study proprioceptive and tactile sensory stimuli during physical finger-object interactions. Conceived for artificial grasp, manipulation, and haptic exploration, the BairClaw could also be used for future studies on the neurorehabilitation of somatosensory disorders due to upper limb impairment or loss. A remote actuation system enables the modular control of tendon-driven hands. The artificial proprioception system enables direct measurement of joint angles and tendon tensions while temperature, vibration, and skin deformation are provided by a multimodal tactile sensor. The provision of multimodal sensory feedback that is spatiotemporally consistent with commanded actions could lead to benefits such as reduced phantom limb pain, and increased prosthesis use due to improved functionality and reduced cognitive burden

Ecological psychology is radical enough: A reply to radical enactivists

Ecological psychology is one of the most influential theories of perception in the embodied, anti-representational, and situated cognitive sciences. However, radical enactivists claim that Gibsonians tend to describe ecological information and its ‘pick up’ in ways that make ecological psychology close to representational theories of perception and cognition. Motivated by worries about the tenability of classical views of informational content and its processing, these authors claim that ecological psychology needs to be “RECtified” so as to explicitly resist representational readings. In this paper, we argue against this call for RECtification. To do so, we offer a detailed analysis of the notion of perceptual information and other related notions such as specificity and meaning, as they are presented in the specialized ecological literature. We defend that these notions, if properly understood, remain free of any representational commitment. Ecological psychology, we conclude, does not need to be RECtified

Nanoscale all-oxide-heterostructured bio-inspired optoresponsive nociceptor

Retina nociceptor, as a key sensory receptor, not only enables the transport of warning signals to the human central nervous system upon its exposure to noxious stimuli, but also triggers the motor response that minimizes potential sensitization. In this study, the capability of two-dimensional all-oxide-heterostructured artificial nociceptor as a single device with tunable properties was confirmed. Newly designed nociceptors utilize ultra-thin sub-stoichiometric TiO2-Ga2O3 heterostructures, where the thermally annealed Ga2O3 films play the role of charge transfer controlling component. It is discovered that the phase transformation in Ga2O3 is accompanied by substantial jump in conductivity, induced by thermally assisted internal redox reaction of Ga2O3 nanostructure during annealing. It is also experimentally confirmed that the charge transfer in all-oxide heterostructures can be tuned and controlled by the heterointerfaces manipulation. Results demonstrate that the engineering of heterointerfaces of two-dimensional (2D) films enables the fabrication of either high-sensitive TiO2-Ga2O3 (Ar) or high-threshold TiO2-Ga2O3 (N-2) nociceptors. The hypersensitive nociceptor mimics the functionalities of corneal nociceptors of human eye, whereas the delayed reaction of nociceptor is similar to high-threshold nociceptive characteristics of human sensory system. The long-term stability of 2D nociceptors demonstrates the capability of heterointerfaces engineering for effective control of charge transfer at 2D heterostructured devices

Visual echolocation concept for the colorophone sensory substitution device using virtual reality

Detecting characteristics of 3D scenes is considered one of the biggest challenges for visually impaired people. This ability is nonetheless crucial for orientation and navigation in the natural environment. Although there are several Electronic Travel Aids aiming at enhancing orientation and mobility for the blind, only a few of them combine passing both 2D and 3D information, including colour. Moreover, existing devices either focus on a small part of an image or allow interpretation of a mere few points in the field of view. Here, we propose a concept of visual echolocation with integrated colour sonification as an extension of Colorophone - an assistive device for visually impaired people. The concept aims at mimicking the process of echolocation and thus provides 2D, 3D and additionally colour information of the whole scene. Even though the final implementation will be realised by a 3D camera, it is first simulated, as a proof of concept, by using VIRCO - a Virtual Reality training and evaluation system for Colorophone. The first experiments showed that it is possible to sonify colour and distance of the whole scene, which opens up a possibility to implement the developed algorithm on a hardware-based stereo camera platform. An introductory user evaluation of the system has been conducted in order to assess the effectiveness of the proposed solution for perceiving distance, position and colour of the objects placed in Virtual Reality