Multimodal perception of histological images for persons blind or visually impaired

Currently there is no suitable substitute technology to enable blind or visually impaired (BVI) people to interpret visual scientific data commonly generated during lab experimentation in real time, such as performing light microscopy, spectrometry, and observing chemical reactions. This reliance upon visual interpretation of scientific data certainly impedes students and scientists that are BVI from advancing in careers in medicine, biology, chemistry, and other scientific fields. To address this challenge, a real-time multimodal image perception system is developed to transform standard laboratory blood smear images for persons with BVI to perceive, employing a combination of auditory, haptic, and vibrotactile feedbacks. These sensory feedbacks are used to convey visual information through alternative perceptual channels, thus creating a palette of multimodal, sensorial information. A Bayesian network is developed to characterize images through two groups of features of interest: primary and peripheral features. Causal relation links were established between these two groups of features. Then, a method was conceived for optimal matching between primary features and sensory modalities. Experimental results confirmed this real-time approach of higher accuracy in recognizing and analyzing objects within images compared to tactile images

An empirical evaluation of a graphics creation technique for blind and visually impaired individuals

The representation of pictorial data by people who are blind and sight impaired has gathered momentum with research and development; however, little research has focused on the use of a screen layout to provide people who are blind and sight impaired users with the spatial orientation to create and reuse graphics. This article contributes an approach to navigating on the screen, manipulating computer graphics, and user-defined images. The technique described in this article enables features such as zooming, grouping, and drawing by calling primitive and user-defined shapes. It enables blind people to engage in and experience drawing and art production on their own. The navigation technique gives an initiative sense of autonomy with compass directions, makes it easy to learn, efficient to manipulate shape with a the simple drawing language, and takes less time to complete with system support features. An empirical evaluation was conducted to validate the suitability of the SETUP09 technique and to evaluate the accuracy, and efficiency of the navigation and drawing techniques proposed. The drawing experiment results confirmed high accuracy (88%) and efficiency among blind and visually impaired (BVI) users

An empirical evaluation of a graphics creation technique for blind and visually impaired individuals

The representation of pictorial data by people who are blind and sight impaired has gathered momentum with research and development; however, little research has focused on the use of a screen layout to provide people who are blind and sight impaired users with the spatial orientation to create and reuse graphics. This paper contributes an approach to navigating on the screen, manipulating computer graphics, and user-defined images. The technique described in this paper enables features such as zooming, grouping, and drawing by calling primitive and user-defined shapes. It enables blind people to engage and experience drawing and art production on their own. The navigation technique gives an initiative sense of autonomy with compass directions, makes it easy to learn, efficient to manipulate shape with a the simple drawing language and takes less time to complete with system support features. An empirical evaluation was conducted to validate the suitability of SETUP09 technique and to evaluate the accuracy, efficiency of the navigation and drawing techniques proposed. The drawing experiment results confirmed high accuracy (88%) and efficiency among BVI users

Screen navigation system for visually impaired people

Purpose – The SETUP09 system consists of both navigation and a computer-aided drawing technique for the people who are blind and visually impaired (BVI). The purpose of this paper is to address the need for a screen navigation technique, which can facilitate a user’s ability to produce art, and scientific diagrams electronically, by introducing a compass-based screen navigation method. Design/methodology/approach – BVI computer users were tested using different screen navigation tasks to assess the accuracy and efficiency of this compass-based navigation technique by using a prototype (SETUP09) and tactile paper grid maps. Findings – The results confirmed that the compass-based navigation facilitates higher accuracy in screen-based moving and location recognition with a noticeable reduction in time and effort. Research limitations/implications – Additional improvements such as the addition of a sound layer to the interface, use of hotkeys, braille and user speech inputs are yet to be tested. Social implications – The current lack of suitable and efficient screen navigation technology is a limiting factor for BVI students and computer users in producing diagrams and drawings. This may place limitations on their career progression and life contentment. It is challenging for a BVI person to draw diagrams and art, which are commonly taught in education or used in industry. The compass-based screen navigation system was developed to address BVI users’ need to be able to create such content. Originality/value – A compass-based navigation method enables screen navigation through a formal command language and enables intuitive movement to a screen location using matrix-style compass directions with zoom-in and zoom-out capabilities

A Formal Approach to Computer Aided 2D Graphical Design for Blind People

The growth of computer aided drawing systems for blind people (CADB) has long been recognised and has increased in interest within the assistive technology research area. The representation of pictorial data by blind and visually impaired (BVI) people has recently gathered momentum with research and development; however, a survey of published literature on CADB reveals that only marginal research has been focused on the use of a formal approach for on screen spatial orientation, creation and reuse of graphics artefacts. To realise the full potential of CADB, such systems should possess attributes of usability, spatial navigation and shape creation features without which blind users drawing activities are less likely to be achieved. As a result of this, usable, effective and self-reliant CADB have arisen from new assistive Technology (AT) research. This thesis contributes a novel, abstract, formal approach that facilitates BVI users to navigate on the screen, create computer graphics/diagrams using 2D shapes and user-defined images. Moreover, the research addresses the specific issues involved with user language by formulating specific rules that make BVI user interaction with the drawing effective and easier. The formal approach proposed here is descriptive and it is specified at a level of abstraction above the concrete level of system technologies. The proposed approach is unique in problem modelling and syntheses of an abstract computer-based graphics/drawings using a formal set of user interaction commands. This technology has been applied to enable blind users to independently construct drawings to satisfy their specific needs without recourse to a specific technology and without the intervention of support workers. The specification aims to be the foundation for a system scope, investigation guidelines and user-initiated command-driven interaction. Such an approach will allow system designers and developers to proceed with greater conceptual clarity than it is possible with current technologies that is built on concrete system-driven prototypes. In addition to the scope of the research the proposed model has been verified by various types of blind users who have independently constructed drawings to satisfy their specific needs without the intervention of support workers. The effectiveness and usability of the proposed approach has been compared against conventional non-command driven drawing systems by different types of blind users. The results confirm that the abstract formal approach proposed here using command-driven means in the context of CADB enables greater comprehension by BVI users. The innovation can be used for both educational and training purposes. The research, thereby sustaining the claim that the abstract formal approach taken allows for the greater comprehension of the command-driven means in the context of CADB, and how the specification aid the design of such a system

Machine Learning Methods for Image Analysis in Medical Applications, from Alzheimer\u27s Disease, Brain Tumors, to Assisted Living

Healthcare has progressed greatly nowadays owing to technological advances, where machine learning plays an important role in processing and analyzing a large amount of medical data. This thesis investigates four healthcare-related issues (Alzheimer\u27s disease detection, glioma classification, human fall detection, and obstacle avoidance in prosthetic vision), where the underlying methodologies are associated with machine learning and computer vision. For Alzheimer’s disease (AD) diagnosis, apart from symptoms of patients, Magnetic Resonance Images (MRIs) also play an important role. Inspired by the success of deep learning, a new multi-stream multi-scale Convolutional Neural Network (CNN) architecture is proposed for AD detection from MRIs, where AD features are characterized in both the tissue level and the scale level for improved feature learning. Good classification performance is obtained for AD/NC (normal control) classification with test accuracy 94.74%. In glioma subtype classification, biopsies are usually needed for determining different molecular-based glioma subtypes. We investigate non-invasive glioma subtype prediction from MRIs by using deep learning. A 2D multi-stream CNN architecture is used to learn the features of gliomas from multi-modal MRIs, where the training dataset is enlarged with synthetic brain MRIs generated by pairwise Generative Adversarial Networks (GANs). Test accuracy 88.82% has been achieved for IDH mutation (a molecular-based subtype) prediction. A new deep semi-supervised learning method is also proposed to tackle the problem of missing molecular-related labels in training datasets for improving the performance of glioma classification. In other two applications, we also address video-based human fall detection by using co-saliency-enhanced Recurrent Convolutional Networks (RCNs), as well as obstacle avoidance in prosthetic vision by characterizing obstacle-related video features using a Spiking Neural Network (SNN). These investigations can benefit future research, where artificial intelligence/deep learning may open a new way for real medical applications

Sensory Communication

Contains table of contents for Section 2, an introduction and reports on twelve research projects.National Institutes of Health Grant R01 DC00117National Institutes of Health Grant R01 DC02032National Institutes of Health/National Institute of Deafness and Other Communication Disorders Grant 2 R01 DC00126National Institutes of Health Grant 2 R01 DC00270National Institutes of Health Contract N01 DC-5-2107National Institutes of Health Grant 2 R01 DC00100U.S. Navy - Office of Naval Research Grant N61339-96-K-0002U.S. Navy - Office of Naval Research Grant N61339-96-K-0003U.S. Navy - Office of Naval Research Grant N00014-97-1-0635U.S. Navy - Office of Naval Research Grant N00014-97-1-0655U.S. Navy - Office of Naval Research Subcontract 40167U.S. Navy - Office of Naval Research Grant N00014-96-1-0379U.S. Air Force - Office of Scientific Research Grant F49620-96-1-0202National Institutes of Health Grant RO1 NS33778Massachusetts General Hospital, Center for Innovative Minimally Invasive Therapy Research Fellowship Gran

Long-Term Consequences of Early Eye Enucleation on Audiovisual Processing

A growing body of research shows that complete deprivation of the visual system from the loss of both eyes early in life results in changes in the remaining senses. Is the adaptive plasticity observed in the remaining intact senses also found in response to partial sensory deprivation specifically, the loss of one eye early in life? My dissertation examines evidence of adaptive plasticity following the loss of one eye (unilateral enucleation) early in life. Unilateral eye enucleation is a unique model for examining the consequences of the loss of binocularity since the brain is completely deprived of all visual input from that eye. My dissertation expands our understanding of the long-term effects of losing one eye early in life on the development of audiovisual processing both behaviourally and in terms of the underlying neural representation. The over-arching goal is to better understand neural plasticity as a result of sensory deprivation. To achieve this I conducted seven experiments, divided into 5 experimental chapters, that focus on the behavioural and structural correlates of audiovisual perception in a unique group of adults who lost one eye in the first few years of life. Behavioural data (Chapters II-V) in conjunction with neuroimaging data (Chapter VI) relate structure and function of the auditory, visual and audiovisual systems in this rare patient group allowing a more refined understanding of cross sensory effects of early sensory deprivation. This information contributes to us better understanding how audiovisual information is experienced by people with one eye. This group can be used as a model to learn how to accommodate and maintain the health of less extreme forms of visual deprivation and to promote overall long-term visual health

Multimodal optical systems for clinical oncology

This thesis presents three multimodal optical (light-based) systems designed to improve the capabilities of existing optical modalities for cancer diagnostics and theranostics. Optical diagnostic and therapeutic modalities have seen tremendous success in improving the detection, monitoring, and treatment of cancer. For example, optical spectroscopies can accurately distinguish between healthy and diseased tissues, fluorescence imaging can light up tumours for surgical guidance, and laser systems can treat many epithelial cancers. However, despite these advances, prognoses for many cancers remain poor, positive margin rates following resection remain high, and visual inspection and palpation remain crucial for tumour detection. The synergistic combination of multiple optical modalities, as presented here, offers a promising solution. The first multimodal optical system (Chapter 3) combines Raman spectroscopic diagnostics with photodynamic therapy using a custom-built multimodal optical probe. Crucially, this system demonstrates the feasibility of nanoparticle-free theranostics, which could simplify the clinical translation of cancer theranostic systems without sacrificing diagnostic or therapeutic benefit. The second system (Chapter 4) applies computer vision to Raman spectroscopic diagnostics to achieve spatial spectroscopic diagnostics. It provides an augmented reality display of the surgical field-of-view, overlaying spatially co-registered spectroscopic diagnoses onto imaging data. This enables the translation of Raman spectroscopy from a 1D technique to a 2D diagnostic modality and overcomes the trade-off between diagnostic accuracy and field-of-view that has limited optical systems to date. The final system (Chapter 5) integrates fluorescence imaging and Raman spectroscopy for fluorescence-guided spatial spectroscopic diagnostics. This facilitates macroscopic tumour identification to guide accurate spectroscopic margin delineation, enabling the spectroscopic examination of suspicious lesions across large tissue areas. Together, these multimodal optical systems demonstrate that the integration of multiple optical modalities has potential to improve patient outcomes through enhanced tumour detection and precision-targeted therapies.Open Acces

Neural and behavioral plasticity in olfactory sensory deprivation

The human brain has a remarkable ability to reorganize as a consequence of altered demands. This ability is particularly noticeable when studying the neural effects of complete sensory deprivation. Both structural and functional cerebral reorganization have repeatedly been demonstrated in individuals with sensory deprivation, most evident in cortical regions associated with the processing of the absent sensory modality. Furthermore, sensory deprivation has been linked to altered abilities in remaining sensory modalities, often of a compensatory character. Although anosmia, complete olfactory sensory deprivation, is our most common sensory deprivation, estimated to affect around 5 % of the population, the effects of anosmia on brain and behavior are still poorly understood. The overall aim of this thesis was to investigate how the human brain and behavior are affected by anosmia, with a focus on individuals with congenital (lifelong) sensory deprivation. Specifically, Study I and Study IV assessed potential behavioral and neural multisensory compensatory abilities whereas Study II and Study III assessed potential reorganization beyond the processing of specific stimuli; the latter by determining morphological and resting-state functional connectivity alterations. Integration of information from different sensory modalities leads to a more accurate perception of the world around us, given that our senses provide complementary information. Although an improved ability to extract multisensory information would be of particular relevance to individuals deprived of one sensory modality, multisensory integration has been sparsely studied in relation to sensory deprivation. In Study I, multisensory integration of audio-visual stimuli was assessed in individuals with anosmia using two different experimental tasks. First, individuals with anosmia were better than matched controls in detecting multisensory temporal asynchronies in a simultaneity judgement task. Second, individuals with congenital, but not acquired, anosmia demonstrated indications of an enhanced ability to utilize multisensory information in an object identification task with degraded stimuli. Based on these results, the neural correlates of audio-visual processing and integration were assessed in individuals with congenital anosmia in Study IV. Relative to matched normosmic individuals, individuals with congenital anosmia demonstrated increased activity in established multisensory regions when integrating degraded audio-visual stimuli; however, no compensatory cross-modal processing in olfactory regions was demonstrated. Together, Study I and IV suggest that complete olfactory sensory deprivation is linked to enhanced audio-visual integration performance that might be facilitated by increased processing in multisensory regions. In Study II, whole-brain gray matter morphology was assessed in individuals with congenital anosmia. Both increases and decreases in the orbitofrontal cortex, a region associated with olfaction and sometimes referred to as secondary olfactory cortex, were observed in individuals with congenital anosmia in relation to matched controls. However, in contrast to our expectations, no sensory deprivation-dependent effects were demonstrated in piriform cortex, a region commonly referred to as primary olfactory cortex. Furthermore, Study III revealed an absence of differences in resting-state functional connectivity between individuals withcongenital anosmia and normosmic individuals within the primary olfactory cortex (including piriform cortex) as well as between core olfactory processing regions. In conclusion, the studies presented within this thesis suggest the existence of a potential multisensory compensatory mechanism in individuals with anosmia, but demonstrate a striking lack of morphological and functional alterations in piriform (primary olfactory) cortex. These results demonstrate that complete olfactory deprivation is associated with a distinct neural and behavioral reorganization in some regions but also a clear lack of effects in other regions; the latter underline the clear differences between our senses and suggest that extrapolating from individual senses should be done cautiously