A robust contour detection operator with combined push-pull inhibition and surround suppression

Contour detection is a salient operation in many computer vision applications as it extracts features that are important for distinguishing objects in scenes. It is believed to be a primary role of simple cells in visual cortex of the mammalian brain. Many of such cells receive push-pull inhibition or surround suppression. We propose a computational model that exhibits a combination of these two phenomena. It is based on two existing models, which have been proven to be very effective for contour detection. In particular, we introduce a brain-inspired contour operator that combines push-pull and surround inhibition. It turns out that this combination results in a more effective contour detector, which suppresses texture while keeping the strongest responses to lines and edges, when compared to existing models. The proposed model consists of a Combination of Receptive Field (or CORF) model with push-pull inhibition, extended with surround suppression. We demonstrate the effectiveness of the proposed approach on the RuG and Berkeley benchmark data sets of 40 and 500 images, respectively. The proposed push-pull CORF operator with surround suppression outperforms the one without suppression with high statistical significance

Brain-Inspired Algorithms for Processing of Visual Data

The study of the visual system of the brain has attracted the attention and interest of many neuro-scientists, that derived computational models of some types of neuron that compose it. These findings inspired researchers in image processing and computer vision to deploy such models to solve problems of visual data processing. In this paper, we review approaches for image processing and computer vision, the design of which is based on neuro-scientific findings about the functions of some neurons in the visual cortex. Furthermore, we analyze the connection between the hierarchical organization of the visual system of the brain and the structure of Convolutional Networks (ConvNets). We pay particular attention to the mechanisms of inhibition of the responses of some neurons, which provide the visual system with improved stability to changing input stimuli, and discuss their implementation in image processing operators and in ConvNets.</p

Computational Modeling of Human Dorsal Pathway for Motion Processing

Reliable motion estimation in videos is of crucial importance for background iden- tification, object tracking, action recognition, event analysis, self-navigation, etc. Re- constructing the motion field in the 2D image plane is very challenging, due to variations in image quality, scene geometry, lighting condition, and most importantly, camera jit- tering. Traditional optical flow models assume consistent image brightness and smooth motion field, which are violated by unstable illumination and motion discontinuities that are common in real world videos. To recognize observer (or camera) motion robustly in complex, realistic scenarios, we propose a biologically-inspired motion estimation system to overcome issues posed by real world videos. The bottom-up model is inspired from the infrastructure as well as functionalities of human dorsal pathway, and the hierarchical processing stream can be divided into three stages: 1) spatio-temporal processing for local motion, 2) recogni- tion for global motion patterns (camera motion), and 3) preemptive estimation of object motion. To extract effective and meaningful motion features, we apply a series of steer- able, spatio-temporal filters to detect local motion at different speeds and directions, in a way that\u27s selective of motion velocity. The intermediate response maps are cal- ibrated and combined to estimate dense motion fields in local regions, and then, local motions along two orthogonal axes are aggregated for recognizing planar, radial and circular patterns of global motion. We evaluate the model with an extensive, realistic video database that collected by hand with a mobile device (iPad) and the video content varies in scene geometry, lighting condition, view perspective and depth. We achieved high quality result and demonstrated that this bottom-up model is capable of extracting high-level semantic knowledge regarding self motion in realistic scenes. Once the global motion is known, we segment objects from moving backgrounds by compensating for camera motion. For videos captured with non-stationary cam- eras, we consider global motion as a combination of camera motion (background) and object motion (foreground). To estimate foreground motion, we exploit corollary dis- charge mechanism of biological systems and estimate motion preemptively. Since back- ground motions for each pixel are collectively introduced by camera movements, we apply spatial-temporal averaging to estimate the background motion at pixel level, and the initial estimation of foreground motion is derived by comparing global motion and background motion at multiple spatial levels. The real frame signals are compared with those derived by forward predictions, refining estimations for object motion. This mo- tion detection system is applied to detect objects with cluttered, moving backgrounds and is proved to be efficient in locating independently moving, non-rigid regions. The core contribution of this thesis is the invention of a robust motion estimation system for complicated real world videos, with challenges by real sensor noise, complex natural scenes, variations in illumination and depth, and motion discontinuities. The overall system demonstrates biological plausibility and holds great potential for other applications, such as camera motion removal, heading estimation, obstacle avoidance, route planning, and vision-based navigational assistance, etc

Coding of multivariate stimuli and contextual interactions in the visual cortex

The primary visual cortex (V1) has long been considered the main low level visual analysis area of the brain. The classical view is of a feedfoward system functioning as an edge detector, in which each cell has a receptive field (RF) and a preferred orientation. Whilst intuitive, this view is not the whole story. Although stimuli outside a neuron’s RF do not result in an increased response by themselves, they do modulate a neuron’s response to what’s inside its RF. We will refer to such extra-RF effects as contextual modulation. Contextual modulation is thought to underlie several perceptual phenomena, such as various orientation illusions and saliency of specific features (such as a contour or differing element). This gives a view of V1 as more than a collection of edge detectors, with neurons collectively extracting information beyond their RFs. However, many of the accounts linking psychophysics and physiology explain only a small subset of the illusions and saliency effects: we would like to find a common principle. So first, we assume the contextual modulations experienced by V1 neurons is determined by the elastica model, which describes the shape of the smoothest curve between two points. This single assumption gives rise to a wide range of known contextual modulation and psychophysical effects. Next, we consider the more general problem of encoding and decoding multi-variate stimuli (such as center surround gratings) in neurons, and how well the stimuli can be decoded under substantial noise levels with a maximum likelihood decoder. Although the maximum likelihood decoder is widely considered optimal and unbiased in the limit of no noise, under higher noise levels it is poorly understood. We show how higher noise levels lead to highly complex decoding distributions even for simple encoding models, which provides several psychophysical predictions. We next incorporate more updated experimental knowledge of contextual modulations. Perhaps the most common form of contextual modulations is center surround modulation. Here, the response to a center grating in the RF is modulated by the presence of a surrounding grating (the surround). Classically this modulation is considered strongest when the surround is aligned with the preferred orientation, but several studies have shown how many neurons instead experience strongest modulation whenever center and surround are aligned. We show how the latter type of modulation gives rise to stronger saliency effects and unbiased encoding of the center. Finally, we take an experimental perspective. Recently, both the presence and the underlying mechanisms of contextual modulations has been increasingly studied in mice using calcium imaging. However, cell signals extracted with calcium imaging are often highly contaminated by other sources. As contextual effects beyond center surround modulation can be subtle, a method is needed to remove the contamination. We present an analysis toolbox to de-contaminate calcium signals with blind source separation. This thesis thus expands our understanding of contextual modulation, predicts several new experimental results, and presents a toolbox to extract signals from calcium imaging data which should allow for more in depth studies of contextual modulation

Brain-Inspired Computing

This open access book constitutes revised selected papers from the 4th International Workshop on Brain-Inspired Computing, BrainComp 2019, held in Cetraro, Italy, in July 2019. The 11 papers presented in this volume were carefully reviewed and selected for inclusion in this book. They deal with research on brain atlasing, multi-scale models and simulation, HPC and data infra-structures for neuroscience as well as artificial and natural neural architectures

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Neural circuits underlying colour vision and visual memory in Drosophila melanogaster

Focusing at the fly visual system I am addressing the identity and function of neurons accomplishing two fundamental processing steps required for survival of most animals: neurons of peripheral circuits underlying colour vision as well neurons of higher order circuits underlying visual memory. Colour vision is commonly assumed to rely on photoreceptors tuned to narrow spectral ranges. In the ommatidium of Drosophila, the four types of so-called inner photoreceptors express different narrow-band opsins. In contrast, the outer photoreceptors have a broadband spectral sensitivity and are thought to exclusively mediate achromatic vision. Using computational models and behavioural experiments, I here demonstrate that the broadband outer photoreceptors contribute to colour vision in Drosophila. A model of opponent processing that includes the opsin of the outer photoreceptors scores the best fit to wavelength discrimination behaviour of flies. To experimentally uncover the contribution of individual photoreceptor types, I used blind flies with disrupted phototransduction (norpA-) and rescued norpA function in genetically targeted photoreceptors and receptor combinations. Surprisingly, dichromatic flies with only broadband photoreceptors and one additional receptor type can discriminate different colours, indicating the existence of a specific output comparison of outer and inner photoreceptors. Furthermore, blocking interneurons postsynaptic to the outer photoreceptors specifically impairs colour but not intensity discrimination. These findings show that outer receptors with a complex and broad spectral sensitivity do contribute to colour vision and reveal that chromatic and achromatic circuits in the fly share common photoreceptors. Higher brain areas integrate sensory input from different modalities including vision and associate these neural representations with good or bad experiences. It is unclear, however, how distinct sensory memories are processed in the Drosophila brain. Furthermore, the neural circuit underlying colour/intensity memory in Drosophila remained so far unknown. In order to address these questions, I established appetitive and aversive visual learning assays for Drosophila. These allow contrasting appetitive and aversive visual memories using neurogenetic methods for circuit analysis. Furthermore, the visual assays are similar to the widely used olfactory learning assays and share reinforcing stimuli (sugar reward and electric shock punishment), conditioning regimes and methods for memory assessment. Thus, a direct comparison of the cellular requirements for visual and olfactory memories becomes feasible. I found that the same subsets of dopamine neurons innervating the mushroom body are necessary and sufficient for formation of both sensory memories. Furthermore, expression of D1-like Dopamine Receptor (DopR) in the mushroom body is sufficient to restore the memory defect of a DopR null mutant (dumb-). These findings and the requirement of the mushroom body for visual memory in the used assay suggest that the mushroom body is a site of convergence, where representations of different sensory modalities may undergo associative modulation.Mit Fokus auf das visuelle System von Fliegen behandle ich in meiner Dissertation die Identität und Funktion von Neuronen, welche zwei fundamentale Verarbeitungsschritte ausführen, die für das Überleben der meisten Tiere notwendig sind. Zum einen sind dies dem Farbensehen zugrunde liegende Neuronen und zum anderen solche, die essentiel für visuelles Gedächtnis sind. Allgemein wird angenommen, dass Farbensehen auf Photorezeptoren mit Sensitivitäten für schmale Spektralbereiche aufbaut. Im Ommatidium von Drosophila exprimieren die sogenannten inneren Photorezeptoren verschiedene spektral schmalbandige Opsine. Im Gegensatz dazu haben die äußeren Photorezeptoren eine breitbandige spektrale Sensitivität und man nimmt an, dass diese ausschließlich achromatisches Sehen ermöglichen. Mit Hilfe von computergestützten Modellen und Verhaltensexperimenten zeige ich hier, dass die breitbandigen äußeren Photorezeptoren zum Farbensehen in Drosophila beitragen. Ein Modell mit opponenter Verarbeitung von Photorezeptorsignalen, welches das Opsin der äußeren Photorezeptoren beinhaltet, passt am besten zum spektralen Unterscheidungsverhalten von Fliegen. Um experimentell den Beitrag der einzelnen Photorezeptortypen zu ermitteln verwendete ich blinde Fliegen mit einem Defekt in der Phototransduktion (norpA-) und rettete die norpA Funktion gezielt in einzelnen oder verschiedenen Kombinationen von Photorezeptortypen mit Hilfe des GAL4/UAS Genexpressionssystems. Erstaunlicherweise können dichromatische Fliegen mit nur äußeren Photorezeptoren und einem weiteren Rezeptortyp Farben unterscheiden, was auf die Existenz eines spezifischen Vergleichs der Signale von äußeren und inneren Photorezeptoren hindeutet. Außerdem beeinträchtigt der Block von Interneuronen, welche postsynaptisch von den äußeren Photorezeptoren sind, spezifisch das Farbensehen aber nicht die Intensitätsunterscheidung. Diese Ergebnisse zeigen zum einen, dass die äußeren Photorezeptoren mit einer komplexen und breitbandigen spektralen Sensitivität zum Farbensehen beitragen und zum anderen, dass chromatische und achromatische neuronale Netzwerke in der Fliege gemeinsame Photorezeptoren teilen. Höher geordnete Gehirnbereiche integrieren sensorische Information verschiedener Modalitäten insbesondere visueller Natur und assoziieren deren neuronale Representation mit guten und schlechten Erfahrungen. Es ist jedoch unklar, wie unterschiedliche sensorische Gedächtnisse im Gehirn von Drosophila verarbeitet werden. Außerdem ist das neuronale Netzwerk, welches Farb- und Intensitätsgedächtnis zugrunde liegt völlig unbekannt. Um diese Fragen zu beantworten etablierte ich appetitive und aversive Verhaltensassays für Drosophila. Diese erlauben die Gegenüberstellung von appetitivem und aversivem visuellen Gedächtnis unter Verwendung von neurogenetischen Methoden zur Netzwerkanalyse. Desweiteren sind die visuellen Verhaltensassays sehr ähnlich zu den verbreiteten olfaktorischen Lernsassays, da diese verstärkende Stimuli (Zuckerbelohnung und Elektroschockbestrafung), Konditionierungsablauf und Methoden zur Gedächtnismessung gemein haben. Dadurch wird ein direkter Vergleich der zellulären Grundlagen von visuellem und olfaktorischem Gedächtnis möglich. Ich fand, dass die gleichen Gruppen von Dopaminneuronen, welche den Pilzkörper innervieren, sowohl notwendig als auch ausreichend für die Bildung beider sensorischer Gedächtnisse sind. Außerdem ist die Expression des D1-ähnlichen Dopaminrezeptors (DopR) im Pilzkörper ausreichend um den Gedächtnisdefekt einer DopR Nullmutante (dumb-) zu retten. Diese Ergebnisse sowie die Notwendigkeit des Pilzkörpers für visuelles Gedächtnis in dem benutzen Assay deuten darauf hin, dass der Pilzkörper ein Konvergenzareal ist, in welchem Repräsentationen von verschiedenen sensorischen Modalitäten assoziativer Modulation unterliegen