Bethge, Matthias

Kümmerer, Matthias

Wallis, Thomas S.A.

English

Purdue E-Pubs

Consistent Saliency Benchmarking: How One Model Can Win on All MetricsMatthias Ku¨mmerer Thomas S.A. Wallis Matthias BethgeUnderstanding how humans place their gaze is important for understanding how humans exlore theirenvironment and for computer vision applications and has attracted research for many decades. Socalled “saliency models” compute a “saliency map” to predict fixations for an image. Many differentsaliency models have been proposed, from low-level feature integration to complex deep-learning basedmodels, and more are added every year.However the field is facing a fundamental problem: there is no agreed-upon metric for assessing thequality of a saliency map. Instead, e.g. the most commonly used MIT saliency benchmark evaluatesa total of eight metrics which yield highly inconsistent model rankings. This has led to contradictingconclusions about which algorithms are most predictive. We have previously shown that treatingmodels probabilistically and evaluating log-density saliency maps removes most of the disagreement,but at the price of performing suboptimally in most metrics.Here we apply Bayesian utility theory to the problem: the metric is the utility function and thesaliency map represents the particular choice one must make given the predicted fixation densityand the utility function. We can show that it is impossible for one saliency map to perform wellin all saliency metrics (Figure 1c). Instead we propose a principled way to derive different metric-dependent saliency maps from a model’s predicted fixation density by maximizing the expected metricperformance of a saliency map under the predicted fixation density (Figure 1b).By converting several influential saliency map models to probabilistic models, we show that ourapproach indeed improves benchmarking on real data. We demonstrate that this allows models tocompete with state-of-the-art for each metric (Figure 2). Furthermore, by using the right saliencymaps for different metrics, the model ranking is consistent over different metrics. This indicates thatthe existing saliency metrics are in fact not measuring different properties of the predicted fixationdensities – they just interpret saliency maps in different ways. This simple yet meaningful way tocompare saliency models should allow a better understanding of which mechanisms work and whichdon’t work and provide a better guide to future research.Figure 1: For a given fixation density (a) we show saliency maps optimal for different metrics (b). Every saliency map performs poorlyin some of the metrics (c).Figure 2: Evaluation on the MIT1003 dataset: The correct saliency map for each metric (thick lines) results in the best possibleperformance among all saliency map types (other lines) for each model and results in consistent model rankings acrossmetrics – unlike the original saliency maps (dashed lines)

Consistent Saliency Benchmarking: How One Model Can Win on All Metrics

https://docs.lib.purdue.edu/cgi/viewcontent.cgi?article=1114&amp;context=modvis

Consistent Saliency Benchmarking: How One Model Can Win on All Metrics

Abstract

Similar works

Full text

Available Versions

Purdue E-Pubs