The mosaic test:benchmarking colour-based image retrieval systems using image mosaics

Evaluation and benchmarking in content-based image retrieval has always been a somewhat neglected research area, making it difficult to judge the efficacy of many presented approaches. In this paper we investigate the issue of benchmarking for colour-based image retrieval systems, which enable users to retrieve images from a database based on lowlevel colour content alone. We argue that current image retrieval evaluation methods are not suited to benchmarking colour-based image retrieval systems, due in main to not allowing users to reflect upon the suitability of retrieved images within the context of a creative project and their reliance on highly subjective ground-truths. As a solution to these issues, the research presented here introduces the Mosaic Test for evaluating colour-based image retrieval systems, in which test-users are asked to create an image mosaic of a predetermined target image, using the colour-based image retrieval system that is being evaluated. We report on our findings from a user study which suggests that the Mosaic Test overcomes the major drawbacks associated with existing image retrieval evaluation methods, by enabling users to reflect upon image selections and automatically measuring image relevance in a way that correlates with the perception of many human assessors. We therefore propose that the Mosaic Test be adopted as a standardised benchmark for evaluating and comparing colour-based image retrieval systems

Vertebra Shape Classification using MLP for Content-Based Image Retrieval

A desirable content-based image retrieval (CBIR) system would classify extracted image features to support some form of semantic retrieval. The Lister Hill National Center for Biomedical Communications, an intramural R&D division of the National Library for Medicine (NLM), maintains an archive of digitized X-rays of the cervical and lumbar spine taken as part of the second national health and nutrition examination survey (NHANES II). It is our goal to provide shape-based access to digitized X-rays including retrieval on automatically detected and classified pathology, e.g., anterior osteophytes. This is done using radius of curvature analysis along the anterior portion, and morphological analysis for quantifying protrusion regions along the vertebra boundary. Experimental results are presented for the classification of 704 cervical spine vertebrae by evaluating the features using a multi-layer perceptron (MLP) based approach. In this paper, we describe the design and current status of the content-based image retrieval (CBIR) system and the role of neural networks in the design of an effective multimedia information retrieval system

The mosaic test:measuring the effectiveness of colour-based image retrieval

A variety of content-based image retrieval systems exist which enable users to perform image retrieval based on colour content - i.e., colour-based image retrieval. For the production of media for use in television and film, colour-based image retrieval is useful for retrieving specifically coloured animations, graphics or videos from large databases (by comparing user queries to the colour content of extracted key frames). It is also useful to graphic artists creating realistic computer-generated imagery (CGI). Unfortunately, current methods for evaluating colour-based image retrieval systems have 2 major drawbacks. Firstly, the relevance of images retrieved during the task cannot be measured reliably. Secondly, existing methods do not account for the creative design activity known as reflection-in-action. Consequently, the development and application of novel and potentially more effective colour-based image retrieval approaches, better supporting the large number of users creating media for use in television and film productions, is not possible as their efficacy cannot be reliably measured and compared to existing technologies. As a solution to the problem, this paper introduces the Mosaic Test. The Mosaic Test is a user-based evaluation approach in which participants complete an image mosaic of a predetermined target image, using the colour-based image retrieval system that is being evaluated. In this paper, we introduce the Mosaic Test and report on a user evaluation. The findings of the study reveal that the Mosaic Test overcomes the 2 major drawbacks associated with existing evaluation methods and does not require expert participants

Improving Recall of Browsing Sets in Image Retrieval from a Semiotics Perspective

The purpose of dissertation is to utilize connotative messages for enhancing image retrieval and browsing. By adopting semiotics as a theoretical tool, this study explores problems of image retrieval and proposes an image retrieval model. The semiotics approach conceptually demonstrates that: 1) a fundamental reason for the dissonance between retrieved images and user needs is representation of connotative messages, and 2) the image retrieval model which makes use of denotative index terms is able to facilitate users to browse connotatively related images effectively even when the users' needs are potentially expressed in the form of denotative query. Two experiments are performed for verifying the semiotic-based image retrieval model and evaluating the effectiveness of the model. As data sources, 5,199 records are collected from Artefacts Canada: Humanities by Canadian Heritage Information Network, and the candidate terms of connotation and denotation are extracted from Art & Architecture Thesaurus. The first experiment, by applying term association measures, verifies that the connotative messages of an image can be derived from denotative messages of the image. The second experiment reveals that the association thesaurus which is constructed based on the associations between connotation and denotation facilitates assigning connotative terms to image documents. In addition, the result of relevant judgments presents that the association thesaurus improves the relative recall of retrieved image documents as well as the relative recall of browsing sets. This study concludes that the association thesaurus indicating associations between connotation and denotation is able to improve the accessibility of the connotative messages. The results of the study are hoped to contribute to the conceptual knowledge of image retrieval by providing understandings of connotative messages within an image and to the practical design of image retrieval system by proposing an association thesaurus which can supplement the limitations of the current content-based image retrieval systems (CBIR)

Content-based access to digital video: the Físchlár system and the TREC video track

This short paper presents an overview of the Físchlár system - an operational digital library of several hundred hours of video content at Dublin City University which is used by over 1,000 users daily, for a variety of applications. The paper describes how Físchlár operates and the services that it provides for users. Following that, the second part of the paper gives an outline of the TREC Video Retrieval track, a benchmarking exercise for information retrieval from video content currently in operation, summarising the operational details of how the benchmarking exercise is operating

Evaluating Text-to-Image Matching using Binary Image Selection (BISON)

Providing systems the ability to relate linguistic and visual content is one of the hallmarks of computer vision. Tasks such as text-based image retrieval and image captioning were designed to test this ability but come with evaluation measures that have a high variance or are difficult to interpret. We study an alternative task for systems that match text and images: given a text query, the system is asked to select the image that best matches the query from a pair of semantically similar images. The system's accuracy on this Binary Image SelectiON (BISON) task is interpretable, eliminates the reliability problems of retrieval evaluations, and focuses on the system's ability to understand fine-grained visual structure. We gather a BISON dataset that complements the COCO dataset and use it to evaluate modern text-based image retrieval and image captioning systems. Our results provide novel insights into the performance of these systems. The COCO-BISON dataset and corresponding evaluation code are publicly available from \url{http://hexianghu.com/bison/}

Creating a test collection to evaluate diversity in image retrieval

This paper describes the adaptation of an existing test collection for image retrieval to enable diversity in the results set to be measured. Previous research has shown that a more diverse set of results often satisfies the needs of more users better than standard document rankings. To enable diversity to be quantified, it is necessary to classify images relevant to a given theme to one or more sub-topics or clusters. We describe the challenges in building (as far as we are aware) the first test collection for evaluating diversity in image retrieval. This includes selecting appropriate topics, creating sub-topics, and quantifying the overall effectiveness of a retrieval system. A total of 39 topics were augmented for cluster-based relevance and we also provide an initial analysis of assessor agreement for grouping relevant images into sub-topics or clusters