Egocentric Computer Vision and Machine Learning for Simulated Prosthetic Vision

Las prótesis visuales actuales son capaces de proporcionar percepción visual a personas con cierta ceguera. Sin pasar por la parte dañada del camino visual, la estimulación eléctrica en la retina o en el sistema nervioso provoca percepciones puntuales conocidas como “fosfenos”. Debido a limitaciones fisiológicas y tecnológicas, la información que reciben los pacientes tiene una resolución muy baja y un campo de visión y rango dinámico reducido afectando seriamente la capacidad de la persona para reconocer y navegar en entornos desconocidos. En este contexto, la inclusión de nuevas técnicas de visión por computador es un tema clave activo y abierto. En esta tesis nos centramos especialmente en el problema de desarrollar técnicas para potenciar la información visual que recibe el paciente implantado y proponemos diferentes sistemas de visión protésica simulada para la experimentación.Primero, hemos combinado la salida de dos redes neuronales convolucionales para detectar bordes informativos estructurales y siluetas de objetos. Demostramos cómo se pueden reconocer rápidamente diferentes escenas y objetos incluso en las condiciones restringidas de la visión protésica. Nuestro método es muy adecuado para la comprensión de escenas de interiores comparado con los métodos tradicionales de procesamiento de imágenes utilizados en prótesis visuales.Segundo, presentamos un nuevo sistema de realidad virtual para entornos de visión protésica simulada más realistas usando escenas panorámicas, lo que nos permite estudiar sistemáticamente el rendimiento de la búsqueda y reconocimiento de objetos. Las escenas panorámicas permiten que los sujetos se sientan inmersos en la escena al percibir la escena completa (360 grados).En la tercera contribución demostramos cómo un sistema de navegación de realidad aumentada para visión protésica ayuda al rendimiento de la navegación al reducir el tiempo y la distancia para alcanzar los objetivos, incluso reduciendo significativamente el número de colisiones de obstáculos. Mediante el uso de un algoritmo de planificación de ruta, el sistema encamina al sujeto a través de una ruta más corta y sin obstáculos. Este trabajo está actualmente bajo revisión.En la cuarta contribución, evaluamos la agudeza visual midiendo la influencia del campo de visión con respecto a la resolución espacial en prótesis visuales a través de una pantalla montada en la cabeza. Para ello, usamos la visión protésica simulada en un entorno de realidad virtual para simular la experiencia de la vida real al usar una prótesis de retina. Este trabajo está actualmente bajo revisión.Finalmente, proponemos un modelo de Spiking Neural Network (SNN) que se basa en mecanismos biológicamente plausibles y utiliza un esquema de aprendizaje no supervisado para obtener mejores algoritmos computacionales y mejorar el rendimiento de las prótesis visuales actuales. El modelo SNN propuesto puede hacer uso de la señal de muestreo descendente de la unidad de procesamiento de información de las prótesis retinianas sin pasar por el análisis de imágenes retinianas, proporcionando información útil a los ciegos. Esté trabajo está actualmente en preparación.<br /

Design, modeling and analysis of object localization through acoustical signals for cognitive electronic travel aid for blind people

El objetivo de la tesis consiste en el estudio y análisis de la localización de objetos en el entorno real mediante sonidos, así como la posterior integración y ensayo de un dispositivo real basado en tal técnica y destinado a personas con discapacidad visual. Con el propósito de poder comprender y analizar la localización de objetos se ha realizado un profundo estado de arte sobre los Sistemas de Navegación desarrollados durante las últimas décadas y orientados a personas con distintos grados de discapacidad visual. En el citado estado del arte, se han analizado y estructurado los dispositivos de navegación existentes, clasificándolos de acuerdo con los componentes de adquisición de datos del entorno utilizados. A este respecto, hay que señalar que, hasta el momento, se conocen tres clases de dispositivos de navegación: 'detectores de obstáculos', que se basan en dispositivos de ultrasonidos y sensores instalados en los dispositivos electrónicos de navegación con el objetivo de detectar los objetos que aparecen en el área de trabajo del sistema; 'sensores del entorno' - que tienen como objetivo la detección del objeto y del usuario. Esta clase de dispositivos se instalan en las estaciones de autobús, metro, tren, pasos de peatones etc., de forma que cuando el sensor del usuario penetra en el área de alcance de los sensores instalados en la estación, éstos informan al usuario sobre la presencia de la misma. Asimismo, el sensor del usuario detecta también los medios de transporte que tienen instalado el correspondiente dispositivo basado en láser o ultrasonidos, ofreciendo al usuario información relativa a número de autobús, ruta etc La tercera clase de sistemas electrónicos de navegación son los 'dispositivos de navegación'. Estos elementos se basan en dispositivos GPS, indicando al usuario tanto su locación, como la ruta que debe seguir para llegar a su punto de destino. Tras la primera etapa de elaboración del estaDunai ., L. (2010). Design, modeling and analysis of object localization through acoustical signals for cognitive electronic travel aid for blind people [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/8441Palanci

Influence of cinematic strategies on architectural design. towards a film institute for Durban.

Master’s Degree. University of KwaZulu-Natal, Durban.No abstract in the PDF

Proceedings of the 9th international conference on disability, virtual reality and associated technologies (ICDVRAT 2012)

The proceedings of the conferenc

Perceiving voids: Memory And Sight Afflictions In Contemporary Cinema

My thesis focuses on the perceptive afflictions caused by alteration of the normal biological functioning of sight and memory. These afflictions are related to the redefinition and disgregation of the classical and postclassical cinematographic characters, and affect cinematographic language, establishing a dialectical relation with the filmic image that contaminates our spectatorial perception. In the first chapter I propose a different reading of a few moments in film history, turning points in which a modification of the ordinary sensorial patterns has been introduced. From the German Expressionism to the late authorial experiments of the 60s, there is a sort of hidden history of film that passes through the continuous redefinition of the audience sensory activity. The different perspective upon broadly studied topics leads to the analysis of contemporary cinema: my thesis tries to investigate the reasons that led cinema to continually increase the representation of perceptive afflictions during the last years, and theses “affected” narratives of afflictions and dysfunctions have interesting effects upon so called “normal” perception of the reality surrounding us. The chapters 2 and 3 respectively analyze memory disorders and different dysfunctions of sight: these elements determine alterations in the ‘normal’ and ‘sensory’ perception of reality. They work as narrative factors changing the visual filmic instruments and redefining the role of the subject (and his/her uncertain definition of identity) in contemporary narratives that show how new technologies are profoundly transforming (and enhancing) the perceptive mechanisms involved in our spectatorial activity. In this work I analyze those films that are mostly committed to a clear and readable narration. My study primarily concentrates on American cinema of the last 30 years – with particular attention to popular Hollywood productions – because Hollywood has become the privileged ‘laboratory’ for the negotiation of gaze and images in the contemporary mediascape (while during the classical era experimental and avant-garde cinema were the “place” in which audience experienced the most important redefinitions of the boundaries between different types of mediated perception

Seeing self and world: everyday photography and young male adults with autism spectrum disorder

With digital image technologies proliferating in contemporary visual culture, the ubiquity of photographs suggests people produce, consume and share photographs widely and routinely, in multiple contexts and with different meanings attached to them. Creating these photographs involves decisions, actions and interventions the photographer makes to guide the viewer and convey a particular message. Illuminating the ways in which photography enables one specific, often overlooked group – young male adults with Autism Spectrum Disorders (ASD) – to visually express the ways they see self and the world, this thesis develops a more inclusive understanding of everyday photographic practices. From the literature that has been reviewed for this study, there has been no investigation that offers a systematic and rigorous approach to empirical enquiry in an effort to explore the photographic image-making of young autistic male adults. The area that has been researched extensively is how autistic people perceive gaze patterns and focus on facial expressions in picture communication systems. While recent studies consider photography and analyse visual perception in ASD, there has been little collaborative discussion in the literature that encompasses autistic people’s own everyday photographic image-making and self-reflective thoughts. This study is one of the first to address this knowledge gap. The methodological framework developed for this qualitative investigation includes participatory visual research methods, and positions this study at the intersection of the recent advances in visual methodologies, and participatory creative methods. Using thematic analysis, the study identified key findings across two dimensions of ASD individuals’ photographic image-making; namely, the phenomenological and social dimensions. Participants’ insights were not only deeply fascinating in their own terms, but also challenged dominant assumptions of digital photography. This qualitative study underlines the importance of multiple senses in the act of taking photographs, while expanding an understanding of what constitutes autistic people’s visual and social worlds. The contribution to knowledge of this investigation is to (i) deepen the knowledge of young male adults with ASD and their everyday photographic practices; and (ii) extend the development of visual and creative research methods. Furthermore, working with this specific group sheds light on photographic practices more broadly

Advances in Robot Navigation

Robot navigation includes different interrelated activities such as perception - obtaining and interpreting sensory information; exploration - the strategy that guides the robot to select the next direction to go; mapping - the construction of a spatial representation by using the sensory information perceived; localization - the strategy to estimate the robot position within the spatial map; path planning - the strategy to find a path towards a goal location being optimal or not; and path execution, where motor actions are determined and adapted to environmental changes. This book integrates results from the research work of authors all over the world, addressing the abovementioned activities and analyzing the critical implications of dealing with dynamic environments. Different solutions providing adaptive navigation are taken from nature inspiration, and diverse applications are described in the context of an important field of study: social robotics

Vision périphérique, caractérisation et suppléance de ses fonctions spatiales

The loss of peripheral vision (tunnel vision) leads to numerous deficits, reducing both independence and quality of life. These deficits reflect spatial cognition impairments, and highlight the close relationship between peripheral vision and spatial representations. This thesis has two main objectives: reaching a better understanding of the nature of these relationships, and using the acquired knowledge in order to propose adaptive, performant and innovative assistive devices able to overcome the peripheral loss. At first, I address the role of peripheral vision in egocentric space coding. I collaborated in a psychophysics experiment, involving detection of visual objects placed in peripheral vision. The visual objects formed similar images on the retina and differed only with respect to their egocentric location: either straight-ahead or eccentric with respect to the head/body midline. We found that straight-ahead objects elicit consistently shorter behavioral responses than eccentric objects. I took in charge a second study evaluating the link between the privileged sensory processing of the straight ahead direction and the dynamic of ocular saccades. Comparison between centripetal and centrifugal pro-saccades and anti-saccades revealed that the superior dynamic of centripetal saccades comes from both sensory and oculomotor factors. These works reveal the early integration of both visual and oculomotor signals in peripheral vision, leading to egocentric representations in which the straight ahead direction is highlighted. Secondly, I investigated the influence of peripheral vision in extracting allocentric spatial representations. In order to assess the role of peripheral vision in allocentric coding, I performed a memory-based pointing task toward previously gazed targets, which were briefly superimposed with visual cues placed at different eccentricities. The results showed that visual cues in peripheral (>10°) vision can contribute to the allocentric coding of a fixated target. A complementary experiment showed that these peripheral allocentric cues play a functional role, notably by facilitating visual searches. These works highlight the importance of peripheral vision in extracting functional spatial relationships between distant elements of the visual environment. Finally, I wanted to promote the development of new assistive devices, able to substitute both egocentric and allocentric spatial functions of the peripheral vision. As a proof of concept, I designed and evaluated a tactile interface mounted on wrist, communicating the spatial location of specific objects and facilitating visual search. Results showed that healthy subjects with artificial tunnel vision (10°) were able to increase by three visual search speeds thank to this tactile interface. Similar results were obtained on a glaucoma subject (field of view 10x15°). My multidisciplinary thesis highlights new roles of peripheral vision in spatial representations and proposes an innovative solution to develop assistive device for tunnel vision.La perte de vision périphérique (vision tubulaire) a pour conséquence de nombreux déficits qui réduisent fortement l'autonomie des personnes qui en sont atteintes et par conséquent leur qualité de vie. Nombre de ces déficits témoignent d'une cognition spatiale dégradée mettant en jeu des relations étroites entre vision périphérique et représentations spatiales. Le double objectif de ce travail de thèse consiste à mieux comprendre la nature de ces relations et, sur la base de ces connaissances, contribuer à l'émergence de dispositifs d'assistance plus adaptés et performants, capables de suppléer l'absence de vision périphérique. Dans un premier temps, je me suis intéressé au rôle de la vision périphérique dans la construction de représentations spatiales égocentrées. J'ai d'abord collaboré à une expérience de psychophysique impliquant la détection en vision périphérique de cibles visuelles situées soit droit-devant, soit excentrées par rapport à l'axe du corps. En mesurant les temps de réaction nous avons pu démontrer que les sujets humains répondaient plus rapidement aux cibles présentées droit-devant qu'aux cibles excentrées. J'ai pris en charge une deuxième étude complémentaire portant sur le lien entre ce traitement sensoriel privilégié du droit-devant en vision périphérique et la dynamique des saccades oculaires de recentrage. En comparant les dynamiques de pro-saccades et d'anti-saccades de recentrage ou d' " excentrage ", j'ai pu mettre en évidence que la supériorité dynamique des saccades de recentrage guidées par la vision périphérique reposait à la fois sur des facteurs sensoriels et oculomoteurs. Ces travaux révèlent l'intégration précoce de signaux visuels et oculomoteurs en vision périphérique, pouvant servir à localiser les éléments visuels par rapport à soi et à privilégier le traitement des éléments situés dans l'axe droit-devant. Dans un deuxième temps, mes travaux ont abordé l'influence de la vision périphérique dans les représentations spatiales allocentrées. Pour étudier l'implication de la vision périphérique dans le codage allocentré, j'ai réalisé une expérience impliquant une tâche de pointage vers des cibles visuelles en vision centrale, accompagnées ou non d'indices visuels à différentes distances en vision périphérique. Les résultats obtenus montrent que des indices visuels capturés par la vision périphérique peuvent effectivement contribuer au codage allocentré d'une cible fixée, et ce même lorsque ces indices périphériques doivent être extraits de scène visuelles complexes en moins de 200 ms. Dans une étude complémentaire, j'ai montré que l'utilisation de ces indices allocentrés situés en périphérie avait un véritable rôle fonctionnel, accélérant les recherches visuelles. Ainsi, ces travaux révèlent une implication fonctionnelle forte de la vision périphérique dans l'extraction des relations spatiales entre éléments présents dans l'environnement visuel. J'ai voulu, durant la dernière partie de ma thèse, initier le développement d'un dispositif d'assistance dont la finalité est de suppléer les fonctions spatiales, égocentrées et allocentrées, de la vision périphérique. Comme preuve de concept, mon travail a consisté à concevoir et développer un dispositif tactile placé sur le poignet et capable de communiquer la position spatiale d'objets d'intérêt pour en simplifier leurs recherches visuelles. Les résultats obtenus chez des sujets sains avec un champ visuel artificiellement réduit (10°) montrent que l'interface tactile permet d'accélérer par trois la vitesse de recherche visuelle. Des résultats similaires ont été observés chez une personne atteinte de glaucome (champ visuel de 10x15°). Ma thèse pluridisciplinaire permet d'apporter un nouvel éclairage sur l'implication de la vision périphérique dans la construction de représentations spatiales, et elle propose de nouvelles pistes pour le développement de dispositifs d'assistance adaptés aux personnes atteintes de vision tubulaire

Zapping the Retina - Understanding electrical responsiveness and electrical desensitization in mouse retinal ganglion cells

The field of science and technology has come a long way since the famous 70’s science fiction series “The Six Million Dollar Man,” where a disabled pilot was transformed into a bionic superhero after receiving artificial implants. What was indeed once a science fiction has now turned into a science fact with the development of various electronic devices interfacing the human neurons in the brain, retina, and limbs. One such advancement was the development of retinal implants. Over the past two decades, the field of retinal prosthetics has made significant advancement in restoring functional vision in patients blinded by diseases such as Retinitis pigmentosa (RP) and Age-related macular degeneration (AMD). RP and AMD are the two leading cause of degenerative blindness. While there is still no definitive cure for either of these diseases, various treatment strategies are currently being explored. Of the various options, the most successful one has been the retinal implants. Retinal implants are small microelectrode or photodiode arrays, which are implanted in the eye of a patient, to stimulate the degenerating retina electrically. They are broadly classified into three types depending on the placement ̶ epiretinal (close proximity to retinal ganglion cells, RGCs) , subretinal (close proximity to bipolar cells, BP) and suprachoroidal (close proximity to choroid). While the ongoing human trials have shown promising results, there remains a considerable variability among patients concerning the quality of visual percepts which limits the working potential of these implants. One such limitation often reported by the implanted patients is “ fading” of visual percepts. Fading refers to the limited ability to elicit temporally stable visual percepts. While, this is not a primary concern for epiretinal implants , it is often observed in subretinal and suprachoroidal implants which use the remaining retinal network to control the temporal spiking pattern of the ganglion cells. The neural correlate of fading is often referred to as “electrical desensitization”, which is the reduction of ganglion cell responses to repetitive electrical stimulation . While much is known about the temporal component of desensitization ( time constant, τ), the spatial aspects (space constant, λ) has not been well characterized. Further, how both these aspects interact to generate spiking responses, remains poorly understood. These crucial questions formed the critical components of my thesis. To address these questions, we stimulated the retinal network electrically, with voltage and current pulses and recorded the corresponding spiking activity using the microelectrode arrays (MEAs). While addressing the primary question of my thesis, we were able to address few idiosyncrasies which has currently stymied the field of retinal prosthetics. At a conceptual level, we have developed an experimental and analysis framework by which one can identify the single stimulus that will activate the most ganglion cells (Chapter 2, Part 1). This stimulus is optimal for ‘blind’ experiments where the specific response properties of each cell are unknown. Furthermore, we attempted to understand the correspondence between the electrical response patterns and visual response types (Chapter 2, Part2). In Chapter 3, we sought to understand better how the visual responses parameters change during ongoing electrical stimulation. We demonstrated that apart from the adaptation occurring due to visual stimulation and invitro experimental conditions, the electrical stimulation alters the RGC visual responses, suggesting the requirement for stimulation-induced changes to be incorporated in the designing of stimulation paradigms for the implant. Finally addressing the primary question (Chapter 4) of my thesis with which we started, we were able to demonstrate, that the electrical desensitization requires the interaction of both time and distance and is not a global phenomenon of the retina. In the final chapter (Chapter 5) we summarize the results of the thesis, discuss the key outcomes and its relevance to the prosthetic field and other vision restoration strategies and the potential future directions of this research. Therefore, in future, to improve the efficacy of retinal prostheses and patient outcomes, it is crucial to have an in-depth understanding of the responsiveness of the retinal circuitry to electrical stimulation

Complexity, the auditory system, and perceptual learning in naïve users of a visual-to-auditory sensory substitution device.

PhDSensory substitution devices are a non-invasive visual prostheses that use sound or touch to aid functioning in the blind. Algorithms informed by natural crossmodal correspondences convert and transmit sensory information attributed to an impaired modality back to the user via an unimpaired modality and utilise multisensory networks to activate visual areas of cortex. While behavioural success has been demonstrated in non-visual tasks suing SSDs how they utilise a metamodal brain, organised for function is still a question in research. While imaging studies have shown activation of visual cortex in trained users it is likely that naïve users rely on auditory characteristics of the output signal for functionality and that it is perceptual learning that facilitates crossmodal plasticity. In this thesis I investigated visual-to-auditory sensory substitution in naïve sighted users to assess whether signal complexity and processing in the auditory system facilitates and limits simple recognition tasks. In four experiments evaluating; signal complexity, object resolution, harmonic interference and information load I demonstrate above chance performance in naïve users in all tasks, an increase in generalized learning, limitations in recognition due to principles of auditory scene analysis and capacity limits that hinder performance. Results are looked at from both theoretical and applied perspectives with solutions designed to further inform theory on a multisensory perceptual brain and provide effective training to aid visual rehabilitation.Queen Mary University of Londo