Subjective evaluation of image understanding results

International audienceImage understanding has many applications. Given an image and a ground truth, it is possible to measure the quality of understanding results provided by different algorithms or parameters. In this paper, we ask some users to make a subjective evaluation of image understanding results by sorting them from the best to the worst. We compared the results with some provided by a metric we defined recently. Experimental results show the good behavior of this metric compared to the human judgment

Parametrization of an image understanding quality metric with a subjective evaluation

International audienceImage understanding has many real industrial applications (video-monitoring, image retrieval . . . ). Given an image and an associated ground truth, it is possible to quantify the quality of understanding results provided by different algorithms or parameters. To this end, it is necessary to take into account many factors for each object in the image: localization and recognition errors and under or over-detection of objects. In order to define an evaluation metric for quantifying the quality of an image understanding result, we have to set, as for example, the weights of each kind of error in the global score. For a correct parameters setting of an evaluation metric we defined previously, we conducted a subjective evaluation of image understanding results involving many experts in image processing. We present in this paper the developed method and analyze the obtained results to weight the various errors in an appropriate way. We show the benefit of this kind of study to define the correct parameters of the metric in order to have a judgment as reliable the one provided by experts. Experimental results on many images from the PASCAL VOC Challenge show the good behavior of this metric compared to the human judgment