Contour Integration in Artifical Neural Networks

Under difficult viewing conditions, the brain's visual system uses a variety of modulatory techniques to supplement its core feedforward signal. One such technique is contour integration, whereby contextual stimuli from outside the classically defined receptive fields of neurons can affect their responses. It manifests in the primary visual (V1) cortex, a low layer of the visual cortex, and can selectively enhance smooth contours. Several mechanistic models, that can account for many of its neurophysiological properties, have been proposed in the literature. However, there has been limited exploration of the learning of biologically realistic contour integration circuits or of the role of contour integration in the processing of natural images. In this thesis, I present a biologically-inspired model of contour integration embedded in a task-driven artificial neural network. The model can relate the low-level neural phenomenon of contour integration to the high-level goals of its encompassing system. It uses intra-area lateral connections and an internal architecture inspired by the V1 cortex. Its parameters are learnt from optimizing performance on high-level tasks rather than being fixed at initialization. When trained to detect contours in a background of random edges, a task commonly used to examine contour integration in the brain, the model learns to integrate contours in a manner consistent with the brain. This is validated by comparing the model with observed data at the behavioral, neurophysiological and neuroanatomical levels. The model is also used to explore the role of contour integration in the perception of natural scenes. I investigate which natural image tasks benefit from contour integration, how it affects their performances and the consistency of trained models with properties of contour integration from more extensively studied artificial stimuli. Specifically, the model was trained on two natural image tasks: detection of all edges and the ability to distinguish if two points lie on the same or different contours. In natural images, the model was found to enhance weaker contours and demonstrated many properties that were similar to when it was trained on synthetic stimuli. Moreover, the features it learnt were robust and generalized well to test data from outside the distribution of training data. The results provide new evidence that contour integration can improve visual perception and complex scene understanding

Perceptual Learning, Long-Range Horizontal Connections And Top-Down Influences In Primary Visual Cortex

The earliest cortical stage of visual processing, the primary visual cortex, has long been seen as a static preprocessor that finds local edges and their orientation like a linear filter bank, and passes this information on to downstream visual areas. This view has been challenged in recent years since the discovery of contextual influences, that is, interactions between the responses of neurons that encode for non-overlapping adjacent areas of visual space, and their anatomical substrate, long-range horizontal connections. These contextual interactions have been shown in awake behaving primates to be modulated depending on the task the animals are performing. A first set of electrophysiological experiments has shown with the help of information theory that when an animal performed one of two tasks on the same visual display, the contextual modulations of the task-relevant parts of the visual display contained more information about the stimulus position than when the same elements were task-irrelevant. A second set of experiments on contour integration was analyzed with ROC analysis to show that an ideal observer could predict the presence of an embedded contour from the spike count of a single neuron on a single trial as well as the animal’s behavioral performance. A final set of experiments showed that prior to learning the same contour integration task, the responses did not contain any information about the stimulus position, that the information in the response increased in parallel with the animals performance during learning, and that the enhanced response after learning disappeared during anesthesia, but is only weakened when performing an irrelevant task in a different part of visual space. Last, a neural network is presented that allows gating of long-range horizontal connections by top-down feedback. The stability and the dynamic behavior of the network have been established with phase-plane analysis. Large-scale simulations have been performed to confirm the stability and show the enhanced contour integration of realistic stimuli as a function of feedback gain. This model has fit quantitatively the electrophysiological experiments of contour integration

Contour Integration : Attentional Effects in a Psychophysics Task and Feature Interactions in a Computational Model

In order to achieve object recognition and image segmentation, the visual system is tasked with combining colinear and cocircular edge configurations into coherent percepts. This process is called Contour integration (CI). CI is believed to be a fundamental visual process, psychophysical experiments have shown humans to be remarkably good at integrating contours even when parts of the contours are occluded, or when a contour does not follow a smooth path. Electrophysiological studies have characterized the neural substrates of contour integration. Based on this information, modelling studies have produced algorithms to explain the functioning of putative mechanisms which give rise to CI. In this thesis, two case studies on contour integration are presented. In the first, psychophysical methods were employed to further characterize humans ability to detect contours under conditions of ambiguity. In particular, this study introduced a novel method in order to determine whether humans remarkable efficiency in detecting contours carries over to dynamic scenes. This is an important question given that scenes in nature are highly dynamic, and up to this point, most CI studies have characterized this process in static scenes. It has often been assumed that CI is a stimulus driven process which leads to pop-out percepts. Results from this study challenge these views. They indicate that humans ability to detect contours deteriorate drastically when shown extended presentations of dynamic stimuli. Furthermore, a set of sub-experiments indicates that top-down processes may play an important role in supporting contour integration under conditions of ambiguity. In the second case study, a computational model of contour integration was developed in order to account for new psychophysical findings, and further understand the mechanisms underlying these observations. Through a number of psychophysical studies, spatial frequency has been shown to be an important feature on which contours can defined and detected, and which can interact with the process of integrating oriented elements. Thus, a modulation component was added to a structurally simple model of contour integration in order to reproduce these findings. The modulation was based on the assumption that interactions of feature detectors are stronger if their preferred spatial frequencies are similar, rather than dissimilar. Extensive numerical simulations were carried out in order to understand the mechanisms leading to the mentioned psychophysical observations, and to reproduce said psychophysical results. This thesis presents contributions to the field of contour integration in two areas. In psychophysics, not only do the results from the experiments reported provide support for the emerging idea that CI may be supported by top-down process, but a significant methodological contribution was also made. A new technique to study CI was introduced. This will allow future research to characterize contour integration under new conditions. In the modeling field, a gap was bridged. To the knowledge of the author, the model presented in this thesis is the first to account for the geometrical characteristics of stimuli and the spatial frequency component of elements in the stimuli

Contour Integration Across Polarities and Spatial Gaps: From Local Contrast Filtering to Global Grouping

This article introduces an experimental paradigm to selectively probe the multiple levels of visual processing that influence the formation of object contours, perceptual boundaries, and illusory contours. The experiments test the assumption that, to integrate contour information across space and contrast sign, a spatially short-range filtering process that is sensitive to contrast polarity inputs to a spatially long-range grouping process that pools signals from opposite contrast polarities. The stimuli consisted of thin subthreshold lines, flashed upon gaps between collinear inducers which potentially enable the formation of illusory contours. The subthreshold lines were composed of one or more segments with opposite contrast polarities. The polarity nearest to the inducers was varied to differentially excite the short-range filtering process. The experimental results are consistent with neurophysiological evidence for cortical mechanisms of contour processing and with the Boundary Contour System model, which identifies the short-range filtering process with cortical simple cells, and the long-range grouping process with cortical bipole cells.Office of Naval Research (N00014-95-1-0409, N00014-95-1-0657); Centre National de la Recherche Scientifique (France) URA (1939

EEG findings of reduced neural synchronization during visual integration in schizophrenia

Schizophrenia patients exhibit well-documented visual processing deficits. One area of disruption is visual integration, the ability to form global objects from local elements. However, most studies of visual integration in schizophrenia have been conducted in the context of an active attention task, which may influence the findings. In this study we examined visual integration using electroencephalography (EEG) in a passive task to elucidate neural mechanisms associated with poor visual integration. Forty-six schizophrenia patients and 30 healthy controls had EEG recorded while passively viewing figures comprised of real, illusory, or no contours. We examined visual P100, N100, and P200 event-related potential (ERP) components, as well as neural synchronization in the gamma (30-60 Hz) band assessed by the EEG phase locking factor (PLF). The N100 was significantly larger to illusory vs. no contour, and illusory vs. real contour stimuli while the P200 was larger only to real vs. illusory stimuli; there were no significant interactions with group. Compared to controls, patients failed to show increased phase locking to illusory versus no contours between 40-60 Hz. Also, controls, but not patients, had larger PLF between 30-40 Hz when viewing real vs. illusory contours. Finally, the positive symptom factor of the BPRS was negatively correlated with PLF values between 40-60 Hz to illusory stimuli, and with PLF between 30-40 Hz to real contour stimuli. These results suggest that the pattern of results across visual processing conditions is similar in patients and controls. However, patients have deficits in neural synchronization in the gamma range during basic processing of illusory contours when attentional demand is limited

Spiking Dynamics during Perceptual Grouping in the Laminar Circuits of Visual Cortex

Grouping of collinear boundary contours is a fundamental process during visual perception. Illusory contour completion vividly illustrates how stable perceptual boundaries interpolate between pairs of contour inducers, but do not extrapolate from a single inducer. Neural models have simulated how perceptual grouping occurs in laminar visual cortical circuits. These models predicted the existence of grouping cells that obey a bipole property whereby grouping can occur inwardly between pairs or greater numbers of similarly oriented and co-axial inducers, but not outwardly from individual inducers. These models have not, however, incorporated spiking dynamics. Perceptual grouping is a challenge for spiking cells because its properties of collinear facilitation and analog sensitivity to inducer configurations occur despite irregularities in spike timing across all the interacting cells. Other models have demonstrated spiking dynamics in laminar neocortical circuits, but not how perceptual grouping occurs. The current model begins to unify these two modeling streams by implementing a laminar cortical network of spiking cells whose intracellular temporal dynamics interact with recurrent intercellular spiking interactions to quantitatively simulate data from neurophysiological experiments about perceptual grouping, the structure of non-classical visual receptive fields, and gamma oscillations.CELEST, an NSF Science of Learning Center (SBE-0354378); SyNAPSE program of the Defense Advanced Research Project Agency (HR001109-03-0001); Defense Advanced Research Project Agency (HR001-09-C-0011

Cortical spatio-temporal dimensionality reduction for visual grouping

The visual systems of many mammals, including humans, is able to integrate the geometric information of visual stimuli and to perform cognitive tasks already at the first stages of the cortical processing. This is thought to be the result of a combination of mechanisms, which include feature extraction at single cell level and geometric processing by means of cells connectivity. We present a geometric model of such connectivities in the space of detected features associated to spatio-temporal visual stimuli, and show how they can be used to obtain low-level object segmentation. The main idea is that of defining a spectral clustering procedure with anisotropic affinities over datasets consisting of embeddings of the visual stimuli into higher dimensional spaces. Neural plausibility of the proposed arguments will be discussed

Neural dynamics of feedforward and feedback processing in figure-ground segregation

Determining whether a region belongs to the interior or exterior of a shape (figure-ground segregation) is a core competency of the primate brain, yet the underlying mechanisms are not well understood. Many models assume that figure-ground segregation occurs by assembling progressively more complex representations through feedforward connections, with feedback playing only a modulatory role. We present a dynamical model of figure-ground segregation in the primate ventral stream wherein feedback plays a crucial role in disambiguating a figure's interior and exterior. We introduce a processing strategy whereby jitter in RF center locations and variation in RF sizes is exploited to enhance and suppress neural activity inside and outside of figures, respectively. Feedforward projections emanate from units that model cells in V4 known to respond to the curvature of boundary contours (curved contour cells), and feedback projections from units predicted to exist in IT that strategically group neurons with different RF sizes and RF center locations (teardrop cells). Neurons (convex cells) that preferentially respond when centered on a figure dynamically balance feedforward (bottom-up) information and feedback from higher visual areas. The activation is enhanced when an interior portion of a figure is in the RF via feedback from units that detect closure in the boundary contours of a figure. Our model produces maximal activity along the medial axis of well-known figures with and without concavities, and inside algorithmically generated shapes. Our results suggest that the dynamic balancing of feedforward signals with the specific feedback mechanisms proposed by the model is crucial for figure-ground segregation

Learning long-range spatial dependencies with horizontal gated-recurrent units

Progress in deep learning has spawned great successes in many engineering applications. As a prime example, convolutional neural networks, a type of feedforward neural networks, are now approaching -- and sometimes even surpassing -- human accuracy on a variety of visual recognition tasks. Here, however, we show that these neural networks and their recent extensions struggle in recognition tasks where co-dependent visual features must be detected over long spatial ranges. We introduce the horizontal gated-recurrent unit (hGRU) to learn intrinsic horizontal connections -- both within and across feature columns. We demonstrate that a single hGRU layer matches or outperforms all tested feedforward hierarchical baselines including state-of-the-art architectures which have orders of magnitude more free parameters. We further discuss the biological plausibility of the hGRU in comparison to anatomical data from the visual cortex as well as human behavioral data on a classic contour detection task.Comment: Published at NeurIPS 2018 https://papers.nips.cc/paper/7300-learning-long-range-spatial-dependencies-with-horizontal-gated-recurrent-unit