A method for evaluating the performance of content-based image retrieval systems based on subjectively determined similarity between images

In recent years multimedia researchers have attempted to design content-based image retrieval systems. However, despite the development of these systems, the term "content " has still remained rather ill defined, and this has made the evaluation of such systems problematic. This paper proposes a method for the creation of a test set of images in which the similarity of each image pair is estimated by two independent methods – by the subjective evaluation of human observers, and by the use of “content words ” as basis vectors that allow the multidimensional content of each image to be represented with a content vector. The latter method is shown to correlate with the subjective judgement of human observers, and thus provides both a more objective method for evaluating and expressing image content, and a possible path to automating the process in the future. 1