Multimodal perception of histological images for persons blind or visually impaired

Currently there is no suitable substitute technology to enable blind or visually impaired (BVI) people to interpret visual scientific data commonly generated during lab experimentation in real time, such as performing light microscopy, spectrometry, and observing chemical reactions. This reliance upon visual interpretation of scientific data certainly impedes students and scientists that are BVI from advancing in careers in medicine, biology, chemistry, and other scientific fields. To address this challenge, a real-time multimodal image perception system is developed to transform standard laboratory blood smear images for persons with BVI to perceive, employing a combination of auditory, haptic, and vibrotactile feedbacks. These sensory feedbacks are used to convey visual information through alternative perceptual channels, thus creating a palette of multimodal, sensorial information. A Bayesian network is developed to characterize images through two groups of features of interest: primary and peripheral features. Causal relation links were established between these two groups of features. Then, a method was conceived for optimal matching between primary features and sensory modalities. Experimental results confirmed this real-time approach of higher accuracy in recognizing and analyzing objects within images compared to tactile images

An empirical evaluation of a graphics creation technique for blind and visually impaired individuals

The representation of pictorial data by people who are blind and sight impaired has gathered momentum with research and development; however, little research has focused on the use of a screen layout to provide people who are blind and sight impaired users with the spatial orientation to create and reuse graphics. This article contributes an approach to navigating on the screen, manipulating computer graphics, and user-defined images. The technique described in this article enables features such as zooming, grouping, and drawing by calling primitive and user-defined shapes. It enables blind people to engage in and experience drawing and art production on their own. The navigation technique gives an initiative sense of autonomy with compass directions, makes it easy to learn, efficient to manipulate shape with a the simple drawing language, and takes less time to complete with system support features. An empirical evaluation was conducted to validate the suitability of the SETUP09 technique and to evaluate the accuracy, and efficiency of the navigation and drawing techniques proposed. The drawing experiment results confirmed high accuracy (88%) and efficiency among blind and visually impaired (BVI) users

An empirical evaluation of a graphics creation technique for blind and visually impaired individuals

The representation of pictorial data by people who are blind and sight impaired has gathered momentum with research and development; however, little research has focused on the use of a screen layout to provide people who are blind and sight impaired users with the spatial orientation to create and reuse graphics. This paper contributes an approach to navigating on the screen, manipulating computer graphics, and user-defined images. The technique described in this paper enables features such as zooming, grouping, and drawing by calling primitive and user-defined shapes. It enables blind people to engage and experience drawing and art production on their own. The navigation technique gives an initiative sense of autonomy with compass directions, makes it easy to learn, efficient to manipulate shape with a the simple drawing language and takes less time to complete with system support features. An empirical evaluation was conducted to validate the suitability of SETUP09 technique and to evaluate the accuracy, efficiency of the navigation and drawing techniques proposed. The drawing experiment results confirmed high accuracy (88%) and efficiency among BVI users

A Formal Approach to Computer Aided 2D Graphical Design for Blind People

The growth of computer aided drawing systems for blind people (CADB) has long been recognised and has increased in interest within the assistive technology research area. The representation of pictorial data by blind and visually impaired (BVI) people has recently gathered momentum with research and development; however, a survey of published literature on CADB reveals that only marginal research has been focused on the use of a formal approach for on screen spatial orientation, creation and reuse of graphics artefacts. To realise the full potential of CADB, such systems should possess attributes of usability, spatial navigation and shape creation features without which blind users drawing activities are less likely to be achieved. As a result of this, usable, effective and self-reliant CADB have arisen from new assistive Technology (AT) research. This thesis contributes a novel, abstract, formal approach that facilitates BVI users to navigate on the screen, create computer graphics/diagrams using 2D shapes and user-defined images. Moreover, the research addresses the specific issues involved with user language by formulating specific rules that make BVI user interaction with the drawing effective and easier. The formal approach proposed here is descriptive and it is specified at a level of abstraction above the concrete level of system technologies. The proposed approach is unique in problem modelling and syntheses of an abstract computer-based graphics/drawings using a formal set of user interaction commands. This technology has been applied to enable blind users to independently construct drawings to satisfy their specific needs without recourse to a specific technology and without the intervention of support workers. The specification aims to be the foundation for a system scope, investigation guidelines and user-initiated command-driven interaction. Such an approach will allow system designers and developers to proceed with greater conceptual clarity than it is possible with current technologies that is built on concrete system-driven prototypes. In addition to the scope of the research the proposed model has been verified by various types of blind users who have independently constructed drawings to satisfy their specific needs without the intervention of support workers. The effectiveness and usability of the proposed approach has been compared against conventional non-command driven drawing systems by different types of blind users. The results confirm that the abstract formal approach proposed here using command-driven means in the context of CADB enables greater comprehension by BVI users. The innovation can be used for both educational and training purposes. The research, thereby sustaining the claim that the abstract formal approach taken allows for the greater comprehension of the command-driven means in the context of CADB, and how the specification aid the design of such a system

Functional and anatomical brain networks : Brain networks during naturalistic auditory stimuli, tactile stimuli and rest : Functional network plasticity in early-blind subjects.

Hearing is a versatile sense allowing us, among other things, to avoid danger and engage in pleasurable discussions. The ease with which we follow a conversation in a noisy environment is astonishing. Study I in this thesis used functional magnetic resonance imaging to explore the large-scale organization of speech and non-speech sound processing during a naturalistic stimulus comprised of an audio drama. Two large-scale functional networks processed the audio drama; one processed only speech, the other processed both speech and non-speech sounds. Hearing is essential for blind subjects. Anatomical and functional changes in the brains of blind people allow them to experience a detailed auditory world, compensating for the lack of vision. Therefore, comparing early-blind subjects brains to those of sighted people during naturalistic stimuli reveals fundamental differences in brain organization. In Study II, naturalistic stimuli were employed to explore whether one of the most distinguishing traits of the auditory system the left-lateralized responses to speech changes following blindness. As expected, in sighted subjects, speech processing was left-hemisphere dominant. Curiously, the left-hemisphere dominance for speech was absent or even reversed in blind subjects. In early-blind people, the senses beyond vision are strained as they try to compensate for the loss of sight; on the other hand, the occipital cortices are devoid of normal visual information flow. Interestingly, in blind people, senses other than vision recruit the occipital cortex. Additional to changes in the occipital cortex, the sensory cortices devoted to touch and hearing change. Data presented here suggested more inter-subject variability in auditory and parietal areas in blind subjects compared with sighted subjects. The study suggested that the greater the inter-subject variability of the network, the greater the experience-dependent plasticity of that network. As the prefrontal areas display large inter-subject spatial variability, the activation of the prefrontal cortex varies greatly. The variable activation might partly explain why the top-down influences of the prefrontal cortex on tactile discrimination are not well understood. In the fourth study, anatomical variability was assessed on an individual level, and transcranial magnetic stimulation was targeted at individually-chosen prefrontal locations indicated in tactile processing. Stimulation of one out of two prefrontal cortex locations impaired the subjects ability to distinguish a single tactile pulse from paired pulses. Thus, the study suggested that tactile information is regulated by functionally specialized prefrontal subareas.Anatomiska och funktionella hjärnnätverk hos seende och synskadade Hörseln är möjligen vårt viktigaste sinne. Med hjälp av hörseln kan vi bl.a. undvika faror och enkelt skilja mellan tal och oljud. I den första delen av avhandlingen fick försökspersonerna lyssna till en radiopjäs medan de undergick funktionell magnetresonanstomografi. Vi fann två olika hjärnnätverk som behandlade pjäsen. Det ena reagerade på både tal och andra ljud, medan det andra registrerade enbart tal. Synskadade klarar sig väl i vardagslivet, detta anses påvisa hjärnans förmåga att anpassa sig till begränsningar i dess normala funktion. I andra delen av avhandlingen jämfördes sedan ungdomen synskadade med seende. Det framgick att seende reagerade starkare på tal med den vänstra hjärnhalvan. Tidigt synskadade visade inga skillnader mellan hjärnhalvorna i reaktivitet till tal. Hos blinda rör det sig naturligtvis inte synintryck i syncortex. Detta medför att tidigt i livet synskadade brukar syncortex för att bearbeta hörsel- och känselintryck. Funktionell magnetresonanstomografi har möjliggjort att undersöka hur synskadade personers hjärnor förändrat sig. I tredje delen av avhandlingen påvisades att synskadade hade en större individuella skillnader i tal- och hörselcortex. Syncortex däremot hade större individuella skillnader hos seende. Det kan tyda på att ökningen av de individuella skillnaderna speglar hjärnans adaptionsmekanismer. En orsak till vår bristfälliga kunskap on frontallobens sammansättning kan ligga i de stora individuella variationerna. Vätskediffusions-MR kan påvisa anatomiska nervbanor på en individuell nivå. I avhandlingens fjärde del utvärderades anatomiska kontakter på individnivå. Sedan användes transkraniell magnetstimulering för att aktivera utvalda nervbanor mellan frontalloben och känslocortex medan vi observerade försökspersonernas förmåga att särskilja antalet känslostötar. Försökspersoner kunde inte urskilja mängden känslostötar ifall ett visst område av frontalloben stimulerades.Sokeiden ja näkevien aivoverkot Tämän väitöskirjatyön tavoitteena oli selvittää aivoverkkojen ominaisuuksia levossa, sekä tunto- ja ääniärsykkeiden aikana. Tarkemmat tavoitteet olivat seuraavat: 1) Tutkia, miten luonnonmukainen ääniärsyke käsitellään näkevien ja sokeiden koehenkilöiden aivoverkoissa. 2) Tutkia toiminnallisten aivoverkkojen yksilöllisiä eroja sokeilla ja näkevillä. 3) Selvittää miten etuaivolohkon ja tuntoaivokuoren välisen hermorata-yhteyden ärsyttäminen vaikuttaa tuntoärsykkeen käsittelyyn. Kuulo on mahdollisesti tärkein aistimme. Kuuloaistin avulla mm. vältämme vaaroja ja erotamme vaivattomasti puheen ympäröivästä melusta. Tämän väitöksen ensimmäisessä osatyössä tutkittavat kuuntelivat ääninäytelmää aivojen toiminnallisen magneettikuvauksen aikana. Tutkimuksessa havaittiin kaksi erillistä ääninäytelmää käsittelevää toiminnallista aivoverkkoa. Toinen käsitteli puheääniä ja ympäristöääniä, toinen pelkästään puheääniä. Sokeat pärjäävät hyvin arkielämässä. Tätä pidetään osoituksena aivojen kyvystä paikata puuttuvaa näköaistia muovautumalla. Toisessa osatyössä tutkittiin, miten varhain sokeutuneiden aivot käsittelevät ääninäytelmää. Kun varhain sokeutuneiden koehenkilöiden aivojen toimintaa verrattiin näkevien aivojen toimintaan, paljastui, että näkevillä vasen aivopuolisko reagoi puheeseen oikeata aivopuoliskoa vahvemmin, kun taas sokeilla tämä aivopuoliskojen välinen ero ei tullut esille. Sokeilla kuuloaivokuori korvaa näköaistin puutetta, toisaalta näköaivokuorelle ei tule näköaistin tuottamia viestejä. Tästä seuraa, että varhain sokeutuneilla muut aistit valtaavat näköaivokuoren. Aivojen toiminnallinen magneettikuvaus sekä uudet analyysimenetelmät ovat mahdollistaneet sokeiden aivojen muovautumisen tutkimisen. Tässä väitöskirjatyössä käytettiin uusia analyysimenetelmiä sokeiden ja näkevien aivokuoren toiminnan muovautumisen tutkimiseen. Osoitimme, että sokeilla oli näkeviä enemmän yksilöllistä vaihtelua puhe- ja kuuloaivokuoren toiminnassa. Näköaivokuoren toiminnassa sokeilla oli vähemmän yksilöllistä vaihtelua kuin näkevillä. Päättelimme, että yksilöllisen vaihtelevuuden lisääntyminen oli osoitus sokeuden seurauksena tapahtuneesta aivojen muovautumisesta. Yhtenä syynä etuaivolohkon järjestäytymisen puutteelliseen tuntemiseen voi olla etuaivolohkon toiminnan suuri yksilöllinen vaihtelevuus. Diffuusiopainotteinen aivokuvantaminen yhdistettynä traktografiaan paljastaa anatomisia hermoratayhteyksiä yksilötasolla. Neljännessä osatyössä arvioimme näitä menetelmiä käyttäen tuntoaivokuoren ja etuaivolohkon anatomisia yhteyksiä yksilötasolla, ja ärsytimme näitä kohteita käyttäen nk. transkraniaalista magneettistimulaatiota (TMS). Kun otsalohkon toimintaa näin häirittiin yksilöllisesti tarkasti suunnatulla TMS pulssilla, tutkittavien kyky eritellä tuntopulssien määrää häiriintyi. Työ osoitti, että otsalohkon etuosa säätelee tuntoaivokuoren toimintaa

Understanding space by moving through it: neural networks of motion- and space processing in humans

Humans explore the world by moving in it, whether moving their whole body as during walking or driving a car, or moving their arm to explore the immediate environment. During movement, self-motion cues arise from the sensorimotor system comprising vestibular, proprioceptive, visual and motor cues, which provide information about direction and speed of the movement. Such cues allow the body to keep track of its location while it moves through space. Sensorimotor signals providing self-motion information can therefore serve as a source for spatial processing in the brain. This thesis is an inquiry into human brain systems of movement and motion processing in a number of different sensory and motor modalities using functional magnetic resonance imaging (fMRI). By characterizing connections between these systems and the spatial representation system in the brain, this thesis investigated how humans understand space by moving through it. In the first study of this thesis, the recollection networks of whole-body movement were explored. Brain activation was measured during the retrieval of active and passive self-motion and retrieval of observing another person performing these tasks. Primary sensorimotor areas dominated the recollection network of active movement, while higher association areas in parietal and mid-occipital cortex were recruited during the recollection of passive transport. Common to both self-motion conditions were bilateral activations in the posterior medial temporal lobe (MTL). No MTL activations were observed during recollection of movement observation. Considering that on a behavioral level, both active and passive self-motion provide sufficient information for spatial estimations, the common activation in MTL might represent the common physiological substrate for such estimations. The second study investigated processing in the 'parahippocampal place area' (PPA), a region in the posterior MTL, during haptic exploration of spatial layout. The PPA in known to respond strongly to visuo-spatial layout. The study explored if this region is processing visuo-spatial layout specifically or spatial layout in general, independent from the encoding sensory modality. In both a cohort of sighted and blind participants, activation patterns in PPA were measured while participants haptically explored the spatial layout of model scenes or the shape of information-matched objects. Both in sighted and blind individuals, PPA activity was greater during layout exploration than during object-shape exploration. While PPA activity in the sighted could also be caused by a transformation of haptic information into a mental visual image of the layout, two points speak against this: Firstly, no increase in connectivity between the visual cortex and the PPA were observed, which would be expected if visual imagery took place. Secondly, blind participates, who cannot resort to visual imagery, showed the same pattern of PPA activity. Together, these results suggest that the PPA processes spatial layout information independent from the encoding modality. The third and last study addressed error accumulation in motion processing on different levels of the visual system. Using novel analysis methods of fMRI data, possible links between physiological properties in hMT+ and V1 and inter-individual differences in perceptual performance were explored. A correlation between noise characteristics and performance score was found in hMT+ but not V1. Better performance correlated with greater signal variability in hMT+. Though neurophysiological variability is traditionally seen as detrimental for behavioral accuracy, the results of this thesis contribute to the increasing evidence which suggests the opposite: that more efficient processing under certain circumstances can be related to more noise in neurophysiological signals. In summary, the results of this doctoral thesis contribute to our current understanding of motion and movement processing in the brain and its interface with spatial processing networks. The posterior MTL appears to be a key region for both self-motion and spatial processing. The results further indicate that physiological characteristics on the level of category-specific processing but not primary encoding reflect behavioral judgments on motion. This thesis also makes methodological contributions to the field of neuroimaging: it was found that the analysis of signal variability is a good gauge for analysing inter-individual physiological differences, while superior head-movement correction techniques have to be developed before pattern classification can be used to this end

Advanced Sensing and Image Processing Techniques for Healthcare Applications

This Special Issue aims to attract the latest research and findings in the design, development and experimentation of healthcare-related technologies. This includes, but is not limited to, using novel sensing, imaging, data processing, machine learning, and artificially intelligent devices and algorithms to assist/monitor the elderly, patients, and the disabled population

Nature of crossmodal plasticity in the blind brain and interplay with sight restoration

Thèse réalisée en cotutelle avec l'Université catholique de Louvain.Ce travail de thèse s’est intéressé à la plasticité cérébrale associée à la privation/restauration visuelle. A travers deux études transversales utilisant l’imagerie par résonance magnétique fonctionnelle auprès d’un groupe de participants présentant une cécité congénitale ou précoce (ainsi qu’auprès d’un groupe contrôle de participants voyants), nous avons tenté de caractériser la manière dont le cortex occipital - typiquement dédié au traitement de l’information visuelle - se réorganise afin de traiter différents stimuli auditifs. Nous démontrons qu’en cas de cécité précoce, différentes régions du cortex occipital présentent une préférence fonctionnelle pour certains types de stimuli non-visuels, avec une spécialisation fonctionnelle qui respecte celle de régions typiquement impliquées dans le traitement d’informations similaires en vision. Ces découvertes constituent une avancée conceptuelle concernant le rôle joué par les contraintes intrinsèques d’une part, et par l’expérience d’autre part, dans l’émergence de réponses sensorielles et fonctionnelles du cortex occipital. D’une part, l’observation de réponses occipitales à la stimulation auditive chez le non-voyant précoce (réorganisation transmodale) rend compte de la capacité du cortex occipital à réorienter sa modalité sensorielle préférentielle en fonction de l’expérience. D’autre part, l’existence de modules cognitifs spécialisés dans le cortex occipital du non-voyant précoce, semblables à ceux du cerveau voyant, démontre les contraintes intrinsèques imposées à une telle plasticité. Dans une étude de cas longitudinale, nous avons également exploré comment les changements plastiques associés à la cécité interagissent avec une récupération visuelle partielle à l’âge adulte. Nous avons réalisé des mesures pré et post-opératoires auprès d’un patient ayant récupéré la vision, en combinant les techniques comportementales ainsi que de neuroimagerie fonctionnelle et structurelle afin d’investiguer conjointement l’évolution de la réorganisation transmodale et de la récupération des fonctions visuelles à travers le temps. Nous démontrons que les changements structurels et fonctionnels caractérisant le cortex occipital du non-voyant sont partiellement réversibles suite à une récupération visuelle à l’âge adulte. De manière générale, ces recherches témoignent de l’importante adaptabilité du cortex occipital aux prises avec des changements drastiques dans l’expérience visuelle.The present Ph.D. work was dedicated to the study of experience-dependent brain plasticity associated with visual deprivation/restoration. In two cross-sectional studies involving the use of functional magnetic resonance imaging in a group of participants with congenital or early blindness (and in a control group of sighted participants), we attempted to characterize the way the occipital cortex - typically devoted to vision – reorganizes itself in order to process different auditory stimuli. We demonstrate that in case of early visual deprivation, distinct regions of the occipital cortex display a functional preference for specific non-visual attributes, maintaining a functional specialization similar to the one that characterizes the sighted brain. Such studies have shed new light on the role played by intrinsic constraints on the one side, and experience on the other, in shaping the modality- and functional tuning of the occipital cortex. On the one hand, the observation of occipital responses to auditory stimulation (crossmodal plasticity) highlights the ability of the occipital cortex to reorient its preferential tuning towards the preserved sensory modalities as a function of experience. On the other hand, the observation of specialized cognitive modules in the occipital cortex, similar to those observed in the sighted, highlights the intrinsic constraints imposed to such plasticity. In a longitudinal single-case study, we further explored how the neuroplastic changes associated with blindness may interact with the newly reacquired visual inputs following partial visual restoration in adulthood. We performed both pre- and post-surgery measurements in a sight-recovery patient combining behavioral, neurostructural and neurofunctional methods in order to jointly investigate the evolution of crossmodal reorganization and visual recovery across time. We demonstrate that functional and structural changes evidenced in the visually-deprived occipital cortex can only partially reverse following sight restoration in adulthood. Altogether, our findings demonstrate the striking adaptability of the occipital cortex facing drastic changes in visual experience

Slaying the chimera: a complementarity approach to the extended mind thesis

Much of the literature directed at the Extended Mind Thesis (EMT) has revolved around parity issues, focussing on the problem of how to individuate the functional roles and on the relevance of these roles for the production of human intelligent behaviour. Proponents of EMT have famously claimed that we shouldn’t take the location of a process as a reliable indicator of the mechanisms that support our cognitive behaviour. This functionalist understanding of cognition has however been challenged by opponents of EMT [such as Rupert (2009); Adams & Aizawa (2009)], who have claimed that differences between internal, biological processes and putatively extended ones not only exist but are actually crucial to undermine the idea that inner and outer are functionally equivalent. This debate about how to individuate the functional roles has led to a treacherous stand-off, in which proponents of EMT have been trapped under the persistent accusation of causal/constitution conflation. My strategy for responding to this charge is to look precisely at those functional differences highlighted by critics of EMT. I reckon that extended cognitive systems are endowed with quite different properties from systems that are “brain bound” and argue that it is precisely these differences that allow human minds to transcend their biological limitations. I thus defend a complementarity version of the extended mind, according to which externally located resources and internal biological elements make a different but complementary contribution to bringing about intelligent behaviour [Sutton (2010)]. My defence of complementarity is based on both the phylogeny and the ontogeny of cognitive systems. I initially explore the interrelation between brain and cognitive development from a neuroconstructivist perspective [Quartz & Sejnowski (1997); Mareshal et al. (2007)] and then argue that our brains do not have fixed functional architectures but are sculpted and given form by the activities we repeatedly engage in. As a result of repeated engagements in socio-cultural tasks, relevant brain pathways undergo substantial rewiring. Development thus scaffolds our brains, which become geared into working in symbiotic partnership with external resources. [Kiverstein & Farina (2011)]. On these grounds, I call into question any tendency to interpret the human biological nature as fixed and endogenously pre-determined and side with proponents of DST [Oyama (2000); Griffiths & Gray (2001)] and ontogenetic niche construction [Stotz (2010)] in arguing that we should think of natural selection as operating on whole developmental systems composed of living organisms in culturally enriched niches. [Wheeler & Clark(2008)]. Complementarity defences of EMT argue that many of the kinds of cognition humans excel at can only be accomplished by brains working together with a body that directly manipulates and acts on the world [Rowlands (2010); Menary (2007)]. I take Sensory Substitution Devices (SSDs henceforth) as my empirical case study. SSDs exploit the remarkable plasticity of our brains and with training supply a novel perceptual modality that compensates for loss or impaired sensory channel. I argue that the coupling with these devices triggers a new mode of phenomenal access to the world, something I propose to label as a kind of “artificial synaesthesia [Ward & Meijer (2010)].This new mode of access to the world transforms our cognitive skills and gives rise to augmented processes of deep bio-technological symbiosis. SSDs therefore become mind enhancing tools [Clark (2003)] and a perfect case study for Complementarity. Having shown the relevance of SSDs for EMT, I then take up the possibility that these devices don’t just relocate the boundaries of cognition but may also stretch the bounds of perceptual awareness. I explore the possibility that perceivers using SSDs count as extended cognitive systems and therefore argue that the experiences they enjoy should be counted as extended conscious experiences.[Kiverstein & Farina, (forthcoming)]. SSDs are quite often said to involve some form of incorporation.[Clark (2008)]. Rupert has challenged this idea and its relevance for EMT on the grounds of his embedded approach. Particularly, he has explained tool-use in terms of the causal interaction between the subject and its detached tool. In the final chapter of my dissertation I critically look at his objections and argue that all his arguments fail to apply to SSDs. In SSD perception in fact the tool becomes geared to work in symbiotic partnership with the active subject and then get factored into its’ body schema so that both of them come to form a single system of cognitive analysis