Saliency identified by absence of background structure

Visual attention is commonly modelled by attempting to characterise objects using features that make them special or in some way distinctive in a scene. These approaches have the disadvantage that it is never certain what features will be relevant in an object that has not been seen before. This paper provides a brief outline of the approaches to modeling human visual attention together with some of the problems that they face. A graphical representation for image similarity is described that relies on the size of maximally associative structures (cliques) that are found to be reflected in pairs of images. While comparing an image with itself, the similarity mechanism is shown to model pop-out effects when constraints are placed on the physical separation of pixels that correspond to nodes in the maximal cliques. Background regions are found to contain structure in common that is not present in the salient regions which are thereby identified by its absence. The approach is illustrated with figures that exemplify asymmetry in pop-out, the conjunction of features, orientation disturbances and the application to natural images

Primary visual cortex as a saliency map: parameter-free prediction of behavior from V1 physiology

It has been hypothesized that neural activities in the primary visual cortex (V1) represent a saliency map of the visual field to exogenously guide attention. This hypothesis has so far provided only qualitative predictions and their confirmations. We report this hypothesis' first quantitative prediction, derived without free parameters, and its confirmation by human behavioral data. The hypothesis provides a direct link between V1 neural responses to a visual location and the saliency of that location to guide attention exogenously. In a visual input containing many bars, one of them saliently different from all the other bars which are identical to each other, saliency at the singleton's location can be measured by the shortness of the reaction time in a visual search task to find the singleton. The hypothesis predicts quantitatively the whole distribution of the reaction times to find a singleton unique in color, orientation, and motion direction from the reaction times to find other types of singletons. The predicted distribution matches the experimentally observed distribution in all six human observers. A requirement for this successful prediction is a data-motivated assumption that V1 lacks neurons tuned simultaneously to color, orientation, and motion direction of visual inputs. Since evidence suggests that extrastriate cortices do have such neurons, we discuss the possibility that the extrastriate cortices play no role in guiding exogenous attention so that they can be devoted to other functional roles like visual decoding or endogenous attention.Comment: 11 figures, 66 page

The left intraparietal sulcus modulates the selection of low salient stimuli

Neuropsychological and functional imaging studies have suggested a general right hemisphere advantage for processing global visual information and a left hemisphere advantage for processing local information. In contrast, a recent transcranial magnetic stimulation study [Mevorach, C., Humphreys, G. W., & Shalev, L. Opposite biases in salience-based selection for the left and right posterior parietal cortex. Nature Neuroscience, 9, 740-742, 2006b] demonstrated that functional lateralization of selection in the parietal cortices on the basis of the relative salience of stimuli might provide an alternative explanation for previous results. In the present study, we applied a whole-brain analysis of the functional magnetic resonance signal when participants responded to either the local or the global levels of hierarchical figures. The task (respond to local or global) was crossed with the saliency of the target level (local salient, global salient) to provide, for the first time, a direct contrast between brain activation related to the stimulus level and that related to relative saliency. We found evidence for lateralization of salience-based selection but not for selection based on the level of processing. Activation along the left intraparietal sulcus (IPS) was found when a low saliency stimulus had to be selected irrespective of its level. A control task showed that this was not simply an effect of task difficulty. The data suggest a specific role for regions along the left IPS in salience-based selection, supporting the argument that previous reports of lateralized responses to local and global stimuli were contaminated by effects of saliency

Distractibility in daily life is reflected in the structure and function of human parietal cortex

We all appreciate that some of our friends and colleagues are more distractible than others. This variability can be captured by pencil and paper questionnaires in which individuals report such cognitive failures in their everyday life. Surprisingly, these self-report measures have high heritability, leading to the hypothesis that distractibility might have a basis in brain structure. In a large sample of healthy adults, we demonstrated that a simple self-report measure of everyday distractibility accurately predicted gray matter volume in a remarkably focal region of left superior parietal cortex. This region must play a causal role in reducing distractibility, because we found that disrupting its function with transcranial magnetic stimulation increased susceptibility to distraction. Finally, we showed that the self-report measure of distractibility reliably predicted our laboratory-based measure of attentional capture. Our findings distinguish a critical mechanism in the human brain causally involved in avoiding distractibility, which, importantly, bridges self-report judgments of cognitive failures in everyday life and a commonly used laboratory measure of distractibility to the structure of the human brai

MASCOT: a mechanism for attention-based scale-invariant object recognition in images

The efficient management of large multimedia databases requires the development of new techniques to process, characterize, and search for multimedia objects. Especially in the case of image data, the rapidly growing amount of documents prohibits a manual description of the images’ content. Instead, the automated characterization is highly desirable to support annotation and retrieval of digital images. However, this is a very complex and still unsolved task. To contribute to a solution of this problem, we have developed a mechanism for recognizing objects in images based on the query by example paradigm. Therefore, the most salient image features of an example image representing the searched object are extracted to obtain a scale-invariant object model. The use of this model provides an efficient and robust strategy for recognizing objects in images independently of their size. Further applications of the mechanism are classical recognition tasks such as scene decomposition or object tracking in video sequences

Why do These Match? Explaining the Behavior of Image Similarity Models

Explaining a deep learning model can help users understand its behavior and allow researchers to discern its shortcomings. Recent work has primarily focused on explaining models for tasks like image classification or visual question answering. In this paper, we introduce Salient Attributes for Network Explanation (SANE) to explain image similarity models, where a model's output is a score measuring the similarity of two inputs rather than a classification score. In this task, an explanation depends on both of the input images, so standard methods do not apply. Our SANE explanations pairs a saliency map identifying important image regions with an attribute that best explains the match. We find that our explanations provide additional information not typically captured by saliency maps alone, and can also improve performance on the classic task of attribute recognition. Our approach's ability to generalize is demonstrated on two datasets from diverse domains, Polyvore Outfits and Animals with Attributes 2. Code available at: https://github.com/VisionLearningGroup/SANEComment: Accepted at ECCV 202