Predicting human gaze using low-level saliency combined with face detection

Under natural viewing conditions, human observers shift their gaze to allocate processing resources to subsets of the visual input. Many computational models try to predict such voluntary eye and attentional shifts. Although the important role of high level stimulus properties (e.g., semantic information) in search stands undisputed, most models are based on low-level image properties. We here demonstrate that a combined model of face detection and low-level saliency significantly outperforms a low-level model in predicting locations humans fixate on, based on eye-movement recordings of humans observing photographs of natural scenes, most of which contained at least one person. Observers, even when not instructed to look for anything particular, fixate on a face with a probability of over 80% within their first two fixations; furthermore, they exhibit more similar scanpaths when faces are present. Remarkably, our model’s predictive performance in images that do not contain faces is not impaired, and is even improved in some cases by spurious face detector responses

Evidence for two distinct mechanisms directing gaze in natural scenes

Various models have been proposed to explain the interplay between bottom-up and top-down mechanisms in driving saccades rapidly to one or a few isolated targets. We investigate this relationship using eye-tracking data from subjects viewing natural scenes to test attentional allocation to high-level objects within a mathematical decision-making framework. We show the existence of two distinct types of bottom-up saliency to objects within a visual scene, which disappear within a few fixations, and modification of this saliency by top-down influences. Our analysis reveals a subpopulation of early saccades, which are capable of accurately fixating salient targets after prior fixation within the same image. These data can be described quantitatively in terms of bottom-up saliency, including an explicit face channel, weighted by top-down influences, determining the mean rate of rise of a decision-making model to a threshold that triggers a saccade. These results are compatible with a rapid subcortical pathway generating accurate saccades to salient targets after analysis by cortical mechanisms

Competition and Attention in the Human Brain Eye-Tracking and Single-Neuron Recordings in Healthy Controls and Individuals with Neurological and Psychiatric Disorders

At any given moment, our brains are bombarded with enormous amounts of information from the environment. External stimuli from the senses travel through our eyes, nose, or skin, and internal reflections and imagery travel from within. All these stimuli are processed in parallel and compete with each other toward one ultimate goal: becoming the single percept which we are aware of at this unique present moment. This work studies the way by which this competition occurs in our brains. The mechanisms and the methods which allow the brain to overcome this load of information by selecting one of many thoughts to reach the upper level of our consciousness. We studied healthy controls in eye-tracking experiments where they viewed sets of images in different tasks. In the first task, subjects freely viewed images with semantic high-level cues such as faces and social scenes. Subjects showed a significant tendency to rapidly attend to the faces in their first fixation. In a second task, subjects viewed similar images in a search task where they were instructed to find objects or faces in the scene. Subjects showed the same tendency to look at faces independent of the task. In a following study, subjects were looking at images with text and phones as control and were shown to look at text rapidly and early, but not as much as faces. Phones, as control, showed no increased rapid attentional attraction. In order to test the magnitude of the effect, we had subjects participate in a third experiment where they were instructed to refrain from looking at these objects, but were not able to do so as easily for faces as they were for text and phones. This suggests an innate mechanism that draws our attention to faces and social scenes early — even in situations where other stimuli compete for our attention. Faces win the competition for our attention regardless of the task. We used this striking result to modify an existing computer model for bottom-up attentional allocation to better predict human’s fixation in images. The results were tested additionally in two groups of individuals with disorders that manifest themselves primarily in decreased social attention: autism and agenesis of the corpus callosum (AgCC; subjects who are missing the bridge between the left and right hemispheres in the brain). Individuals with these disorders were tested in the same paradigms and indeed showed decreased attention allocation to faces or social cues in the images. AgCC subjects showed an even lower level of interest in social cues. While the results suggested a competition for attention that has social cues win over alternative cues in healthy controls, the psychiatric disorder groups show no such effect. In order to test the competition in the brain even further we tested individuals with epilepsy undergoing brain surgery, who were implanted with electrodes to record from single neurons in their medial temporal lobe (MTL). These patients participated in a task where they were projecting their thoughts of one of various concepts onto a computer screen, in real-time, using a decoder that interpreted their thoughts and imagery. Patients performed a task in which they were instructed to think of one of four concepts, and by accurately doing so were fading into an image representation of that concept on the computer screen. Multiple images that were shown on the screen simultaneously while the patient tried to suppress one and maintain an imagery of the other allowed us to directly create a situation where competition between various external stimuli, and in turn multiple brain regions, is tested. Subjects were able to reach high level of control of their single MTL neurons after very little training. In this direct measure of the competition between brain regions and neurons in the brain, we show that attention can be modulated to direct the flow of information to one or the other area, even though the external stimuli from the environment is identical. We used the results from the fading experiment to construct a model of the mechanism by which competition between external stimuli and internal imagery modulates attention in the brain. Subjects who were able to reach an even higher threshold of control of a single neuron played a computer game in which they were controlling an airplane on the screen using their thoughts alone. These subjects reached a high level of control suggesting an ability to use single neurons in the MTL for brain-machine interfaces with very high accuracy. Finally, we report various case studies from experiments involving direct measures of attentional allocation by individuals with face blindness (Prosopagnosia) who performed poorly in tasks involving competition between facial and social attention attractors; a subject with no amygdalae who was unable to direct her attention to fearful entities including images of herself posing while displaying fearful emotions; identical twins who showed an extremely high correlation in their attentional allocation metrics — both eye-tracking and individual rating of interest in images; and a similar high correlation between a mother and her autistic son. Altogether, these results shed light on the processes and the mechanisms undergoing in our brain, in the milliseconds between the moment information starts flowing in our brain from the environment, through the spotlight of attention that selects which of various inputs will be selected to reach our consciousness.</p

The Psychology of Corporate Rights

Relying on the corporate personhood doctrine, the U.S. Supreme Court has increasingly expanded the scope of rights granted to corporations and other forms of collective entities. While this trend has received widespread attention in legal scholarship an

Faces and text attract gaze independent of the task: Experimental data and computer model

Previous studies of eye gaze have shown that when looking at images containing human faces, observers tend to rapidly focus on the facial regions. But is this true of other high-level image features as well? We here investigate the extent to which natural scenes containing faces, text elements, and cell phones - as a suitable control - attract attention by tracking the eye movements of subjects in two types of tasks - free viewing and search. We observed that subjects in free-viewing conditions look at faces and text 16.6 and 11.1 times more than similar regions normalized for size and position of the face and text. In terms of attracting gaze, text is almost as effective as faces. Furthermore, it is difficult to avoid looking at faces and text even when doing so imposes a cost. We also found that subjects took longer in making their initial saccade when they were told to avoid faces/text and their saccades landed on a non-face/non-text object. We refine a well-known bottom–up computer model of saliency-driven attention that includes conspicuity maps for color, orientation, and intensity by adding high-level semantic information (i.e., the location of faces or text) and demonstrate that this significantly improves the ability to predict eye fixations in natural images. Our enhanced model’s predictions yield an area under the ROC curve over 84% for images that contain faces or text when compared against the actual fixation pattern of subjects. This suggests that the primate visual system allocates attention using such an enhanced saliency map

Latency and Selectivity of Single Neurons Indicate Hierarchical Processing in the Human Medial Temporal Lobe

Neurons in the temporal lobe of both monkeys and humans show selective responses to classes of visual stimuli and even to specific individuals. In this study, we investigate the latency and selectivity of visually responsive neurons recorded from microelectrodes in the parahippocampal cortex, entorhinal cortex, hippocampus, and amygdala of human subjects during a visual object presentation task. During 96 experimental sessions in 35 subjects, we recorded from a total of 3278 neurons. Of these units, 398 responded selectively to one or more of the presented stimuli. Mean response latencies were substantially larger than those reported in monkeys. We observed a highly significant correlation between the latency and the selectivity of these neurons: the longer the latency the greater the selectivity. Particularly, parahippocampal neurons were found to respond significantly earlier and less selectively than those in the other three regions. Regional analysis showed significant correlations between latency and selectivity within the parahippocampal cortex, entorhinal cortex, and hippocampus, but not within the amygdala. The later and more selective responses tended to be generated by cells with sparse baseline firing rates and vice versa. Our results provide direct evidence for hierarchical processing of sensory information at the interface between the visual pathway and the limbic system, by which increasingly refined and specific representations of stimulus identity are generated over time along the anatomic pathways of the medial temporal lobe

Comparing social attention in autism and amygdala lesions: Effects of stimulus and task condition

The amygdala plays a critical role in orienting gaze and attention to socially salient stimuli. Previous work has demonstrated that SM a patient with rare bilateral amygdala lesions, fails to fixate and make use of information from the eyes in faces. Amygdala dysfunction has also been implicated as a contributing factor in autism spectrum disorders (ASD), consistent with some reports of reduced eye fixations in ASD. Yet, detailed comparisons between ASD and patients with amygdala lesions have not been undertaken. Here we carried out such a comparison, using eye tracking to complex social scenes that contained faces. We presented participants with three task conditions. In the Neutral task, participants had to determine what kind of room the scene took place in. In the Describe task, participants described the scene. In the Social Attention task, participants inferred where people in the scene were directing their attention. SM spent less time looking at the eyes and much more time looking at the mouths than control subjects, consistent with earlier findings. There was also a trend for the ASD group to spend less time on the eyes, although this depended on the particular image and task. Whereas controls and SM looked more at the eyes when the task required social attention, the ASD group did not. This pattern of impairments suggests that SM looks less at the eyes because of a failure in stimulus-driven attention to social features, whereas individuals with ASD look less at the eyes because they are generally insensitive to socially relevant information and fail to modulate attention as a function of task demands. We conclude that the source of the social attention impairment in ASD may arise upstream from the amygdala, rather than in the amygdala itself

Scene-selective coding by single neurons in the human parahippocampal cortex

Imaging, electrophysiological, and lesion studies have shown a relationship between the parahippocampal cortex (PHC) and the processing of spatial scenes. Our present knowledge of PHC, however, is restricted to the macroscopic properties and dynamics of bulk tissue; the behavior and selectivity of single parahippocampal neurons remains largely unknown. In this study, we analyzed responses from 630 parahippocampal neurons in 24 neurosurgical patients during visual stimulus presentation. We found a spatially clustered subpopulation of scene-selective units with an associated event-related field potential. These units form a population code that is more distributed for scenes than for other stimulus categories, and less sparse than elsewhere in the medial temporal lobe. Our electrophysiological findings provide insight into how individual units give rise to the population response observed with functional imaging in the parahippocampal place area

Selectivity of pyramidal cells and interneurons in the human medial temporal lobe

Neurons in the medial temporal lobe (MTL) respond selectively to pictures of specific individuals, objects, and places. However, the underlying mechanisms leading to such degree of stimulus selectivity are largely unknown. A necessary step to move forward in this direction involves the identification and characterization of the different neuron types present in MTL circuitry. We show that putative principal cells recorded in vivo from the human MTL are more selective than putative interneurons. Furthermore, we report that putative hippocampal pyramidal cells exhibit the highest degree of selectivity within the MTL, reflecting the hierarchical processing of visual information. We interpret these differences in selectivity as a plausible mechanism for generating sparse responses

On-line, voluntary control of human temporal lobe neurons

Daily life continually confronts us with an exuberance of external, sensory stimuli competing with a rich stream of internal deliberations, plans and ruminations. The brain must select one or more of these for further processing. How this competition is resolved across multiple sensory and cognitive regions is not known; nor is it clear how internal thoughts and attention regulate this competition. Recording from single neurons in patients implanted with intracranial electrodes for clinical reasons, here we demonstrate that humans can regulate the activity of their neurons in the medial temporal lobe (MTL) to alter the outcome of the contest between external images and their internal representation. Subjects looked at a hybrid superposition of two images representing familiar individuals, landmarks, objects or animals and had to enhance one image at the expense of the other, competing one. Simultaneously, the spiking activity of their MTL neurons in different subregions and hemispheres was decoded in real time to control the content of the hybrid. Subjects reliably regulated, often on the first trial, the firing rate of their neurons, increasing the rate of some while simultaneously decreasing the rate of others. They did so by focusing onto one image, which gradually became clearer on the computer screen in front of their eyes, and thereby overriding sensory input. On the basis of the firing of these MTL neurons, the dynamics of the competition between visual images in the subject’s mind was visualized on an external display