Smith, Tim J.

Birkbeck Institutional Research Online

Smith, T. J. & Henderson, J. M. (2010) The causal influence of visual salience on gaze guidance during scene search and memorization, In: Proceedings of the 18th Annual Workshop on Object Perception, Attention and Memory; St. Louis, Missouri, USA, 18th November, 2010.   AUTHORS  Tim J. Smith, University of Edinburgh, tim.smith@ed.ac.uk  John M. Henderson, University of South Carolina, jhender@mailbox.sc.edu.   TITLE  The causal influence of visual salience on gaze guidance during scene search and memorization  ABSTRACT   Visual salience at fixation is often reported as significantly higher than chance. However, it is unclear if this gaze behaviour is caused by visual salience as comparisons have always been between objects or scenes. In the present study the salience of a target object was manipulated within scenes. Removing visual salience had a minimal effect on gaze behaviour but increasing saliency significantly increased the probability of early fixation during both search and memorisation. These results suggest that visual salience may play a causal role in fixation probability but its contribution independent from cognitive relevance is currently unknown.   SUMMARY   The guidance of gaze during scene viewing is thought to be influenced by both “bottom-up” visual features such as luminance and edges and “top-down” cognitive factors such as scene context and viewing task. Koch & Ullman (1985) proposed that bottom-up visual features may “pop-out” and capture attention due to the computation of a visual saliency map in the early visual system. The computational implementation of the visual saliency model (Itti & Koch, 2001) has successfully promoted the “bottom-up” guidance of gaze during scene viewing but evidence confirming the salience hypothesis is currently mixed. Bottom-up visual features such as edges and luminance contrast are higher at fixation than control locations during free-viewing (Parkhurst, Law, & Niebur, 2002; Tatler, Baddeley, & Gilchrist, 2005) and medium salience objects are fixated earlier and more often than low-salience objects during memorization (Foulsham & Underwood, 2007). However, the contribution of salience is minimal compared to other cognitive factors such as searching for specific objects (Foulsham & Underwood, 2007; Henderson, Brockmole, Castelhano, & Mack, 2007; Henderson, Malcolm, & Schandl, 2009).  The main problem with identifying the causal contribution of visual salience to gaze guidance is an inherent correlation with other higher-order factors, e.g. semantically meaningful scene regions are also visually salient (Henderson et al., 2007). Comparisons between low and high salience objects are typically performed between different objects or scenes (Foulsham & Underwood, 2007; Henderson et al., 2009), introducing potential confounds. Without controlling other factors that may correlate with salience it is unclear whether any difference in gaze behaviour is caused by salience alone or these higher-order confounds.  The causal role of visual salience in gaze guidance was investigated in the present study by increasing or decreasing the visual salience of a target object in a scene and examining gaze behaviour to the object during both memorization and search. If visual salience influences gaze behaviour then removing saliency should decrease attention to the object and increasing saliency should increase attention. This effect of salience may be most apparent during scene memorization due to the absence of a search template.    METHODOLOGY  Figure 1: Left column = Example scenes used in this study and the corresponding saliency manipulation of the target object (indicated by white rectangle). Right column = Cumulative probability of fixating target object at each fixation during a trial split by saliency of target and viewing task.  Twenty four participants viewed sixty photographs of real-world scenes either for a subsequent memory test (not administered) or to search for a named object within the scene. Scenes were presented for 12 seconds or until a response was made. Eye movements were monitored using an Eyelink 1000.  In order to test the causal impact of visual saliency on gaze behaviour, the saliency of the target object was either removed (Saliency Down) or increased (Saliency Up). To remove salience: 1) the most salient object in the scene was identified via the Matlab implementation of the visual saliency model (Walther & Koch, 2006); 2) the luminance of the entire scene was decreased to 80% of the original using Matlab; 3) the target object was then cut from the low luminance version and pasted into the original image (see Figure 1). The salience of the new target object was then checked to ensure that it had no salience, i.e. was not ranked by the saliency model. In the Salience Up condition the opposite process was used: a non-salient target was identified, its luminance increased to 130% and then pasted back into the original scene. This method of manipulation ensured that all other factors of the scene and object semantics remain the same and only salience is manipulated.   Each participant viewed fifteen scenes in each condition: Naturally Salient, Artificially Non-Salient, Naturally Non-Salient or Artificially Salient target (see Figure 1). Conditions were randomly ordered and balanced for each scene across participants.    RESULTS  The influence of visual salience and task on gaze behaviour was analysed independently within each salience condition.   Saliency Down  The probability of fixating the target object at least once during a trial was significantly greater in Search than Memorization (F(1,22)=16.42, p<.001) but there was no effect of Salience or interaction with task (Search: Natural Salience=0.944, Artificial Non-Salience=0.938; Memory: Natural Salience=0.838, Artificial Non-Salience=0.811). Time to first fixate the target object (expressed as the average ordinal number of the first fixation) was significantly quicker in search compared to memorization (F(1,22)=65.15, p<.001) but there was no effect of Salience or interaction with task (Search: Natural Salience=6.59 fixations, Artificial Non-Salience=6.72; Memory: Natural Salience=12.67, Artificial Non-Salience=13.65). The cumulative probability of fixating the artificially non-salient target was slightly lower than the naturally salient target during memorization although this difference was not significant (see Figure 1, Top). Reducing the salience of the target had no effect on fixation behaviour during search.  Saliency Up  The probability of fixating the target object was significantly greater in Search than Memorization (F(1,22)=11.091, p<.01) but this difference was overridden by the saliency manipulation which increased the fixation probability during memory (Natural Non-Salience=0.78) to the same level (Artificial Salience=0.94) as during search (Natural Non-Salience=0.93, Artificial Salience=0.94) creating a significant main effect of Salience and interaction with Task. Artificial Salience also shortened the time to first fixate the target in both search (Natural Non-Salience=6.44, Artificial Salience=3.61, p<.01) and memory (Natural Non-Salience=15.29, Artificial Salience=9.06, p<.001). The impact of the artificial salience is clearly visible in the cumulative fixation probability (Figure 1, Bottom) which is significantly greater than the natural non-salient condition by the third fixation during both search and memory. Increasing the visual salience of a target object appears to make it “pop-out” of the scene.  DISCUSSION  The present study suggests that naturally occurring visual salience has only marginal (if any) causal influence over gaze guidance during scene memorization and search but an object can be made to “pop-out” of a scene by artificially increasing its salience via luminance contrast. Such bright points may naturally occur in a scene due to reflectance or light sources. However, such points only contribute significantly to gaze behaviour when they co-occur with cognitively relevant features such as foreground objects (Vincent, Baddeley, Correani, Troscianko, & Leonards, 2009). It is currently unclear whether the increase in early fixation of artificially salient objects observed in the present study is due to saliency alone or in combination with the object’s relevance to the search task or scene semantics.   REFERENCES  Foulsham, T., & Underwood, G. (2007). How does the purpose of inspection influence the potency of visual salience in scene perception? Perception, 36, 1123-1138. Henderson, J. M., Brockmole, J. R., Castelhano, M. S., & Mack, M. (2007). Visual saliency does not account for eye movements during visual search in real-world scenes. In R. van Gompel, M. Fischer, W. Murray & R. Hill (Eds.), Eye movements: A window on mind and brain (pp. 537-562). Oxford: Elsevier. Henderson, J. M., Malcolm, G. L., & Schandl, C. (2009). Searching in the dark: Cognitive relevance drives attention in real-world scenes. Psychonomic Bulletin & Review, 16, 850-856. Itti, L., & Koch, C. (2001). Computational modelling of visual attention. Nature Reviews Neuroscience, 2(3), 194-203. Koch, C., & Ullman, S. (1985). Shifts in Selective Visual-Attention - Towards the Underlying Neural Circuitry. Human Neurobiology, 4(4), 219-227. Parkhurst, D., Law, K., & Niebur, E. (2002). Modeling the role of salience in the allocation of overt visual attention. Vision Research, 42(1), 107-123. Tatler, B. W., Baddeley, R. J., & Gilchrist, I. D. (2005). Visual correlates of fixation selection: effects of scale and time. Vision Research, 45(5), 643-659. Vincent, B. T., Baddeley, R. J., Correani, A., Troscianko, T., & Leonards, U. (2009). Do we look at lights? Using mixture modelling to distinguich between low- and high-level factos in natural image viewing. Visual Cognition, 17(6/7), 856-879. Walther, D., & Koch, C. (2006). Modeling attention to salient proto-objects. Neural Networks, 19, 1395-1407.   

The causal influence of visual salience on gaze guidance during scene search and memorization

Abstract

Similar works

Full text

Available Versions

Birkbeck Institutional Research Online