Bethge, Matthias

Kümmerer, Matthias

Wallis, Thomas S.A.

English

Purdue E-Pubs

Selecting Maximally-Predictive Deep Featuresto Explain What Drives Fixations in Free-ViewingMatthias Kümmerer Thomas S.A. Wallis Matthias BethgeRecent advances in deep learning have allowed to predict a substantial amount of the explainableinformation in the spatial fixation distribution in natural images. For example, our model DeepGazeII uses deep features from the VGG deep neural network trained on object recognition as imagerepresentation and combines them in a simple pixelwise nonlinear way to predict a fixation density.However, while these models are very successful at predicting fixations, they are mainly black boxesand therefore not very good at explaining what drives fixations. Here we address this problem byselecting features that are maximally predictive for fixations in a stepwise fashion (Baddeley & Tatler2006). Starting from a version of DeepGaze II without any VGG features (a pure centerbias model),we first search for the VGG feature that maximally improves model performance when added to thismodel. Subsequently, we continue the greedy search strategy by looking for the next feature to addthat yields the biggest improvement. To reduce computational effort of the feature selection, we use asearch strategy that iteratively trains models on subsets of candidate features, selects most promisingsubsets and splits them into smaller partitions. We call this strategy partition search.The search results in an ordered list of features where it can be clearly quantified what additionalcontribution each feature gives to fixation prediction performance while still allowing nonlinear inter-actions between features via the readout architecture of DeepGaze II. A single VGG feature that issensitive to very general popout (including objects) already explains more than half of the performanceof DeepGaze II. Adding a text sensitive feature, a face sensitive feature and a “geometry” feature yieldsa model that uses only 4 VGG features and performs at 85% of the full model (see Figures for details).The deep VGG features we found in this work turned out to have surprisingly intuitive interpre-tations. We are currently extending the model to use features that are interpretable by design, suchas BagNets (Brendel et al. 2019) that allow to determine how localized the features are that drivefixational eye movements.1 feature 2 features 3 features 4 features all 512 features0.00.20.40.60.8information gain [bit/fix](object-)popout+text+faces+"geometry"+otherFigure 1: Fixations can be predicted with high precision using a few interpretable VGG features. Each feature, when added to theprevious model, improves performance by a certain amount (dark green bar) that can be attributed to the patterns the featureis sensitive to. The light green bars show how well the added feature performs when used without any other features: there isa substantial amount of shared information between the features.prediction VGG feature activation previous prediction new predictionVGG feature activationFigure 2: The VGG feature best at predicting fixations is sensitive to a very general form of popout (left) and explains more than halfof the full DeepGaze II performance. The next best feature improves performance to nearly 70% of the full model and issensitive to text (right).

Selecting Maximally-Predictive Deep Features to Explain What Drives Fixations in Free-Viewing

https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1149&amp;context=modvis

Selecting Maximally-Predictive Deep Features to Explain What Drives Fixations in Free-Viewing

Abstract

Similar works

Full text

Available Versions

Purdue E-Pubs