RGB-D Scene Representations for Prosthetic Vision

This thesis presents a new approach to scene representation for prosthetic vision. Structurally salient information from the scene is conveyed through the prosthetic vision display. Given the low resolution and dynamic range of the display, this enables robust identification and reliable interpretation of key structural features that are missed when using standard appearance-based scene representations. Specifically, two different types of salient structure are investigated: salient edge structure, for depiction of scene shape to the user; and salient object structure, for emulation of biological attention deployment when viewing a scene. This thesis proposes and evaluates novel computer vision algorithms for extracting salient edge and salient object structure from RGB-D input. Extraction of salient edge structure from the scene is first investigated through low-level analysis of surface shape. Our approach is based on the observation that regions of irregular surface shape, such as the boundary between the wall and the floor, tend to be more informative of scene structure than uniformly shaped regions. We detect these surface irregularities through multi-scale analysis of iso-disparity contour orientations, providing a real time method that robustly identifies important scene structure. This approach is then extended by using a deep CNN to learn high level information for distinguishing salient edges from structural texture. A novel depth input encoding called the depth surface descriptor (DSD) is presented, which better captures scene geometry that corresponds to salient edges, improving the learned model. These methods provide robust detection of salient edge structure in the scene. The detection of salient object structure is first achieved by noting that salient objects often have contrasting shape from their surroundings. Contrasting shape in the depth image is captured through the proposed histogram of surface orientations (HOSO) feature. This feature is used to modulate depth and colour contrast in a saliency detection framework, improving the precision of saliency seed regions and through this the accuracy of the final detection. After this, a novel formulation of structural saliency is introduced based on the angular measure of local background enclosure (LBE). This formulation addresses fundamental limitations of depth contrast methods and is not reliant on foreground depth contrast in the scene. Saliency is instead measured through the degree to which a candidate patch exhibits foreground structure. The effectiveness of the proposed approach is evaluated through both standard datasets as well as user studies that measure the contribution of structure-based representations. Our methods are found to more effectively measure salient structure in the scene than existing methods. Our approach results in improved performance compared to standard methods during practical use of an implant display

Egocentric Computer Vision and Machine Learning for Simulated Prosthetic Vision

Las prótesis visuales actuales son capaces de proporcionar percepción visual a personas con cierta ceguera. Sin pasar por la parte dañada del camino visual, la estimulación eléctrica en la retina o en el sistema nervioso provoca percepciones puntuales conocidas como “fosfenos”. Debido a limitaciones fisiológicas y tecnológicas, la información que reciben los pacientes tiene una resolución muy baja y un campo de visión y rango dinámico reducido afectando seriamente la capacidad de la persona para reconocer y navegar en entornos desconocidos. En este contexto, la inclusión de nuevas técnicas de visión por computador es un tema clave activo y abierto. En esta tesis nos centramos especialmente en el problema de desarrollar técnicas para potenciar la información visual que recibe el paciente implantado y proponemos diferentes sistemas de visión protésica simulada para la experimentación.Primero, hemos combinado la salida de dos redes neuronales convolucionales para detectar bordes informativos estructurales y siluetas de objetos. Demostramos cómo se pueden reconocer rápidamente diferentes escenas y objetos incluso en las condiciones restringidas de la visión protésica. Nuestro método es muy adecuado para la comprensión de escenas de interiores comparado con los métodos tradicionales de procesamiento de imágenes utilizados en prótesis visuales.Segundo, presentamos un nuevo sistema de realidad virtual para entornos de visión protésica simulada más realistas usando escenas panorámicas, lo que nos permite estudiar sistemáticamente el rendimiento de la búsqueda y reconocimiento de objetos. Las escenas panorámicas permiten que los sujetos se sientan inmersos en la escena al percibir la escena completa (360 grados).En la tercera contribución demostramos cómo un sistema de navegación de realidad aumentada para visión protésica ayuda al rendimiento de la navegación al reducir el tiempo y la distancia para alcanzar los objetivos, incluso reduciendo significativamente el número de colisiones de obstáculos. Mediante el uso de un algoritmo de planificación de ruta, el sistema encamina al sujeto a través de una ruta más corta y sin obstáculos. Este trabajo está actualmente bajo revisión.En la cuarta contribución, evaluamos la agudeza visual midiendo la influencia del campo de visión con respecto a la resolución espacial en prótesis visuales a través de una pantalla montada en la cabeza. Para ello, usamos la visión protésica simulada en un entorno de realidad virtual para simular la experiencia de la vida real al usar una prótesis de retina. Este trabajo está actualmente bajo revisión.Finalmente, proponemos un modelo de Spiking Neural Network (SNN) que se basa en mecanismos biológicamente plausibles y utiliza un esquema de aprendizaje no supervisado para obtener mejores algoritmos computacionales y mejorar el rendimiento de las prótesis visuales actuales. El modelo SNN propuesto puede hacer uso de la señal de muestreo descendente de la unidad de procesamiento de información de las prótesis retinianas sin pasar por el análisis de imágenes retinianas, proporcionando información útil a los ciegos. Esté trabajo está actualmente en preparación.<br /

Real-world indoor mobility with simulated prosthetic vision:The benefits and feasibility of contour-based scene simplification at different phosphene resolutions

Contains fulltext : 246314.pdf (Publisher’s version ) (Open Access)Neuroprosthetic implants are a promising technology for restoring some form of vision in people with visual impairments via electrical neurostimulation in the visual pathway. Although an artificially generated prosthetic percept is relatively limited compared with normal vision, it may provide some elementary perception of the surroundings, re-enabling daily living functionality. For mobility in particular, various studies have investigated the benefits of visual neuroprosthetics in a simulated prosthetic vision paradigm with varying outcomes. The previous literature suggests that scene simplification via image processing, and particularly contour extraction, may potentially improve the mobility performance in a virtual environment. In the current simulation study with sighted participants, we explore both the theoretically attainable benefits of strict scene simplification in an indoor environment by controlling the environmental complexity, as well as the practically achieved improvement with a deep learning-based surface boundary detection implementation compared with traditional edge detection. A simulated electrode resolution of 26 x 26 was found to provide sufficient information for mobility in a simple environment. Our results suggest that, for a lower number of implanted electrodes, the removal of background textures and within-surface gradients may be beneficial in theory. However, the deep learning-based implementation for surface boundary detection did not improve mobility performance in the current study. Furthermore, our findings indicate that, for a greater number of electrodes, the removal of within-surface gradients and background textures may deteriorate, rather than improve, mobility. Therefore, finding a balanced amount of scene simplification requires a careful tradeoff between informativity and interpretability that may depend on the number of implanted electrodes.14 p

Development of Low-Frequency Repetitive Transcranial Magnetic Stimulation as a Tool to Modulate Visual Disorders: Insights from Neuroimaging

Repetitive transcranial magnetic stimulation (rTMS) has become a popular neuromodulation technique, increasingly employed to manage several neurological and psychological conditions. Despite its popular use, the underlying mechanisms of rTMS remain largely unknown, particularly at the visual cortex. Moreover, the application of rTMS to modulate visual-related disorders is under-investigated. The goal of the present research was to address these issues. I employ a multitude of neuroimaging techniques to gain further insight into neural mechanisms underlying low-frequency (1 Hz) rTMS to the visual cortex. In addition, I begin to develop and refine clinical low-frequency rTMS protocols applicable to visual disorders as an alternative therapy where other treatment options are unsuccessful or where there are simply no existing therapies. One such visual disorder that can benefit from rTMS treatment is the perception of visual hallucinations that can occur following visual pathway damage in otherwise cognitively healthy individuals. In Chapters 23, I investigate the potential of multiday low-frequency rTMS to the visual cortex to alleviate continuous and disruptive visual hallucinations consequent to occipital injury. Combining rTMS with magnetic resonance imaging techniques reveals functional and structural cortical changes that lead to the perception of visual hallucinations; and rTMS successfully attenuates these anomalous visual perceptions. In Chapters 45, I compare the effects of alternative doses of low-frequency rTMS to the visual cortex on neurotransmitter levels and intrinsic functional connectivity to gain insight into rTMS mechanisms and establish the most effective protocol. Differential dose-dependent effects are observed on neurotransmitter levels and functional connectivity that suggest the choice of protocol critically depends on the neurophysiological target. Collectively, this work provides a basic framework for the use of low-frequency rTMS and neuroimaging in clinical application for visual disorders

Understanding the temporal dynamics of visual hallucinations in Parkinson's Disease with dementia

PhD ThesisBackground Integrative models of visual hallucinations (VH) posit that the symptom requires disruptions to both bottom-up and top-down visual processing. Although many lines of evidence point to a mixture of aberrant processing and disconnection between key nodes in the visual system, in particular the dorsal and ventral attention networks, there have been no attempts to understand the dynamic behaviour of these systems in Parkinson’s disease with dementia (PDD) with VH. Aims The primary aim of this thesis was to explore the correlates of synaptic communication in the visual system and how spatio-temporal dynamics of the early visual system are altered in relation to the severity of VH. The secondary aim was to help understand the balance between the contributions of bottom-up and top-down processing for the experience of VH in PDD. Methods An assortment of investigative approaches, including resting state electroencephalography (EEG), visual evoked potentials (VEPs), and concurrent EEG and transcranial magnetic stimulation (TMS) were applied in a group of PDD patients with a range of VH severities (n = 26) and contrasted with a group of age matched healthy controls (n = 17). Results Latency of the N1 component was similar between groups, suggesting intact transfer between the retina and the cortex. However, PDD patients had an inherent reduction in the amplitude of the VEP components and displayed a pattern of declining P1 latencies in association with more frequent and severe VH. Evoked potentials arising from TMS of the striate cortex were similar in amplitude and latency for each of the components between PDD and controls. However, inter-component activity at several stages was altered in the PDD group, whilst the frequency and severity of VH was positively associated with the amplitudes of several components in the occipital and parietal regions. Finally, attentional modulation as measured by the alpha-band reactivity was also compromised in PDD patients. iv Conclusions These data provide neurophysiological evidence that both early bottom-up and top-down dysfunctions of the visual system occur in PDD patients who hallucinate, thus supporting integrative models of VH.National Institute for Health Research (NIHR) Biomedical Research Unit (BRU)

The neural processes generating visual phenomenal consciousness: ERP and neuronavigated brain stimulation studies

One of the greatest conundrums to the contemporary science is the relation between consciousness and brain activity, and one of the specifi c questions is how neural activity can generate vivid subjective experiences. Studies focusing on visual consciousness have become essential in solving the empirical questions of consciousness. Th e main aim of this thesis is to clarify the relation between visual consciousness and the neural and electrophysiological processes of the brain. By applying electroencephalography and functional magnetic resonance image-guided transcranial magnetic stimulation (TMS), we investigated the links between conscious perception and attention, the temporal evolution of visual consciousness during stimulus processing, the causal roles of primary visual cortex (V1), visual area 2 (V2) and lateral occipital cortex (LO) in the generation of visual consciousness and also the methodological issues concerning the accuracy of targeting TMS to V1. Th e results showed that the fi rst eff ects of visual consciousness on electrophysiological responses (about 140 ms aft er the stimulus-onset) appeared earlier than the eff ects of selective attention, and also in the unattended condition, suggesting that visual consciousness and selective attention are two independent phenomena which have distinct underlying neural mechanisms. In addition, while it is well known that V1 is necessary for visual awareness, the results of the present thesis suggest that also the abutting visual area V2 is a prerequisite for conscious perception. In our studies, the activation in V2 was necessary for the conscious perception of change in contrast for a shorter period of time than in the case of more detailed conscious perception. We also found that TMS in LO suppressed the conscious perception of object shape when TMS was delivered in two distinct time windows, the latter corresponding with the timing of the ERPs related to the conscious perception of coherent object shape. Th e result supports the view that LO is crucial in conscious perception of object coherency and is likely to be directly involved in the generation of visual consciousness. Furthermore, we found that visual sensations, or phosphenes, elicited by the TMS of V1 were brighter than identically induced phosphenes arising from V2. Th ese fi ndings demonstrate that V1 contributes more to the generation of the sensation of brightness than does V2. Th e results also suggest that top-down activation from V2 to V1 is probably associated with phosphene generation. The results of the methodological study imply that when a commonly used landmark (2 cm above the inion) is used in targeting TMS to V1, the TMS-induced electric fi eld is likely to be highest in dorsal V2. When V1 was targeted according to the individual retinotopic data, the electric fi eld was highest in V1 only in half of the participants. Th is result suggests that if the objective is to study the role of V1 with TMS methodology, at least functional maps of V1 and V2 should be applied with computational model of the TMS-induced electric fi eld in V1 and V2. Finally, the results of this thesis imply that diff erent features of attention contribute diff erently to visual consciousness, and thus, the theoretical model which is built up of the relationship between visual consciousness and attention should acknowledge these diff erences. Future studies should also explore the possibility that visual consciousness consists of several processing stages, each of which have their distinct underlying neural mechanisms.Tajunnallisuus ja sen suhde aivojen neuraalisiin tapahtumiin on yksi tieteen suurimmista ratkaisemattomista kysymyksistä. Tyypillisesti tajunnallisuudella viitataan fenomenaaliseen tajuntaan eli yksilön elämykselliseen ja välittömään kokemukseen tietystä sisällöstä. Tajunnallinen näkeminen eli visuaalinen tajunta on noussut keskiöön tajunnan neuraalisten korrelaattien tutkimuksessa. Tarkastelen tässä tutkimuksessa aivokuoren aktivaation ja visuaalisen tajunnan välistä korrelaatio- ja kausaalisuhdetta elektroenkefalografi an (EEG), toiminnallisten magneettikuvien avulla ohjatun transkraniaalisen magneettistimulaation (TMS) sekä TMS:n indusoiman sähkökentän mallinnuksen avulla. Erityisesti tavoitteena on tarkentaa näönvaraisen tajunnan ja tarkkaavaisuuden välistä suhdetta, tajunnan ajallista kehittymistä, ensimmäisen näköaivokuoren alueen (alue V1), alueen V2 ja lateraalisen näköaivokuoren (LO-alue) roolia visuaalisessa tajunnassa. Väitöskirja koostuu viidestä osatutkimuksesta. Tulokset osoittivat, että varhaisimmat visuaalisen tajunnan vaikutukset tapahtumasidonnaisiin herätevasteisiin (ERP) tulivat esiin noin 140 ms ärsykkeen esittämisen jälkeen ja selvästi ennen valikoivan tarkkaavaisuuden vaikutusta sekä riippumatta valikoivan tarkkaavaisuuden vaikutuksesta. Tulos viittaa siihen, että visuaalisen tajunnan ja valikoivan tarkkaavaisuuden taustalla on erilliset neuraaliset prosessit. Alueen V1 tiedetään olevan välttämätön normaalille näönvaraiselle tajunnalliselle kokemukselle, mutta kolmannen osatutkimuksen tulokset tukevat oletusta, että myös viereinen alue V2 on välttämätön normaalille visuaaliselle tajunnalle. Lisäksi havaittiin, että aktivaatio alueella V2 oli välttämätöntä visuaalisen ärsykkeen yksityiskohtien prosessoinnille pidempään kuin tajunnallisuudelle ärsykkeen läsnäolosta. LO-alueen stimulointi TMS:lla taas ehkäisi tajunnallisen kokemuksen tutusta objektista kahdessa erillisessä aikaikkunassa, joista jälkimmäisen ajoitus korreloi tajuntaan liittyvän tyypillisen ERP-vasteen ajoituksen kanssa. Tutkimustulos tuo tukea näkemykselle jonka mukaan LO-alueen aktivaatio liittyy suoraan niihin prosesseihin, jotka generoivat tajunnallisen havainnon objektista. Okkipitaalilohkon TMS- ja sähköstimuloinnin tiedetään aiheuttavan subjektiivisia valoaistimuksia eli fosfeeneja. Tutkimuksessa havaittiin, että alueen V1 ja alueen V2 stimuloinnin avulla tuotetut fosfeenit ovat keskenään hyvin samankaltaisia muodon, värin sekä koon osalta, mutta alueen V1 stimuloinnissa tuotetut fosfeenit olivat kaikilla tutkittavilla kirkkaampia kuin alueen V2 stimuloinnilla tuotetut fosfeenit. Menetelmällisessä tutkimuksessa havaittiin, että vaikka TMS-pulssi oli suunnattu alueelle V1 toiminnallisten magneettikuvien tai kallon muodon mukaan, oli todennäköisempää, että indusoitu sähkökenttä oli ollut voimakkaampi alueen V2 päällä. Toisaalta toisen osatutkimuksen tulokset osoittivat, että joillekin tutkittaville alueen V1 TMS-stimulaatio oli mahdollista, kun erityistä huomiota kiinnitettiin retinotooppisten edustusalueiden valitsemiseen ja hyödynnettiin sähkökentänmallinnusmenetelmää. Kokonaisuudessaan tämän tutkimuksen tulokset viittaavat siihen, että eri tarkkaavaisuuden muodot vaikuttavat eri tavoin näönvaraiseen tajuntaan, ja näin ollen, teoreettisen mallin visuaalisen tajunnan ja tarkkaavaisuuden välisestä suhteesta tulisi ottaa huomioon nämä erot. Tulevissa tutkimuksissa tulisi myös selvittää mahdollisuutta, jonka mukaan näönvarainen tajunta koostuu useista prosessointi tasosta, joista jokaisella on erilliset hermostolliset perustansa.Siirretty Doriast

Validation of Transcranial Electrical Stimulation (TES) Finite Element Modeling Against MREIT Current Density Imaging in Human Subjects

abstract: Transcranial electrical stimulation (tES) is a non-invasive brain stimulation therapy that has shown potential in improving motor, physiological and cognitive functions in healthy and diseased population. Typical tES procedures involve application of weak current (< 2 mA) to the brain via a pair of large electrodes placed on the scalp. While the therapeutic benefits of tES are promising, the efficacy of tES treatments is limited by the knowledge of how current travels in the brain. It has been assumed that the current density and electric fields are the largest, and thus have the most effect, in brain structures nearby the electrodes. Recent studies using finite element modeling (FEM) have suggested that current patterns in the brain are diffuse and not concentrated in any particular brain structure. Although current flow modeling is useful means of informing tES target optimization, few studies have validated tES FEM models against experimental measurements. MREIT-CDI can be used to recover magnetic flux density caused by current flow in a conducting object. This dissertation reports the first comparisons between experimental data from in-vivo human MREIT-CDI during tES and results from tES FEM using head models derived from the same subjects. First, tES FEM pipelines were verified by confirming FEM predictions agreed with analytic results at the mesh sizes used and that a sufficiently large head extent was modeled to approximate results on human subjects. Second, models were used to predict magnetic flux density, and predicted and MREIT-CDI results were compared to validate and refine modeling outcomes. Finally, models were used to investigate inter-subject variability and biological side effects reported by tES subjects. The study demonstrated good agreements in patterns between magnetic flux distributions from experimental and simulation data. However, the discrepancy in scales between simulation and experimental data suggested that tissue conductivities typically used in tES FEM might be incorrect, and thus performing in-vivo conductivity measurements in humans is desirable. Overall, in-vivo MREIT-CDI in human heads has been established as a validation tool for tES predictions and to study the underlying mechanisms of tES therapies.Dissertation/ThesisDoctoral Dissertation Biomedical Engineering 201

Studying hemispheric asymmetries in visually responsive areas: a TMS-EEG study

The last decades of neuroscientific research have seen a gradual flourishing of studies regarding the neural correlates of human consciousness, with evidence from perceptual studies and theoretical models progressively trying to elucidate the brain dynamics responsible for awareness to emerge. However, despite of the everincreasing number of studies in the field, many aspects are still waiting for clarification. One example of this, in the field of visual awareness, regards the possible hemispheric asymmetry in the neural mechanisms giving rise to visual experiences. In fact, it is known by now that areas located along both the classically defined ventral stream (associated with “vision for perception”) and along the dorsal stream (the “vision for action” stream) can elicit visual percepts – in the form of phosphenes – when stimulated via transcranial magnetic stimulation. However, until now a direct comparison between the two hemispheres in the neural dynamics giving rise to these visual percepts has never been done. With this work, therefore, we tried to shed light on possible differences between the two hemispheres in two cortical areas associated with either one of the two streams: we stimulated the early visual cortex (Experiment 1) and the posterior parietal cortex (Experiment 2) of both hemispheres to elicit phosphenes and compare the associated EEG activity. In both cases we found a clear hemispheric difference, with a left hemisphere showing an early local activation, followed by a more widespread ignition of neural activity; the right hemisphere, on the other side, displayed a later activation mainly localized over central electrodes. These results, consistent across the two experiments, point to the existence of distinct neural mechanisms in the two hemispheres for perceptual awareness. The last part of this work is dedicated to better understand the functioning of transcranial magnetic stimulation, a stimulation technique commonly used in cognitive neuroscience. In spite of its widespread diffusion, the specific influence of some stimulation parameters is not completely understood. To shed some light on this aspect, we stimulated three premotor cortical targets in close proximity, each at three different coil orientation (0°, 45° and 90° respect to stimulated site). Our 4 aim was to disentangle the effect of coil orientation and slight coil transitions on the elicited TEP response. Our preliminary results seem to suggest that both factors have an influence, with orientation being the most influential factor: specifically, an orientation perpendicular to that of the stimulated gyrus seems to be able to elicit the strongest and most reliable response