An information foraging theory based user study of an adaptive user interaction framework for content-based image retrieval

This paper presents the design and results of a task-based user study, based on Information Foraging Theory, on a novel user interaction framework - uInteract - for content-based image retrieval (CBIR). The framework includes a four-factor user interaction model and an interactive interface. The user study involves three focused evaluations, 12 simulated real life search tasks with different complexity levels, 12 comparative systems and 50 subjects. Information Foraging Theory is applied to the user study design and the quantitative data analysis. The systematic findings have not only shown how effective and easy to use the uInteract framework is, but also illustrate the value of Information Foraging Theory for interpreting user interaction with CBIR

Designing interfaces for exploratory content based image retrieval systems

Content Based Image Retrieval or CBIR systems have become the state of the art image retrieval technique over the past few years. They showed commendable retrieval performance over traditional annotation based retrieval. CBIR systems use relevance feedback as input query. CBIR systems developed so far did not put much effort to come up with suitable user interfaces for accepting relevance feedback efficiently i.e. by putting less cognitive load to the user and providing a higher amount of exploration in a limited amount of time. In this study we propose a new interface 'FutureView' which allows peeking into the future providing access to more images in less time than traditional interfaces. This idea helps the user to choose more appropriate images without getting diverted. We used Gaussian process upper confidence bound algorithm for recommending images. We successfully compared this algorithm with Random and Exploitation algorithms with positive results

Content-based image retrieval system for an image gallery search application

Content-based image retrieval is a process framework that applies computer vision techniques for searching and managing large image collections more efficiently. With the growth of large digital image collections triggered by rapid advances in electronic storage capacity and computing power, there is a growing need for devices and computer systems to support efficient browsing, searching, and retrieval for image collections. Hence, the aim of this project is to develop a content-based image retrieval system that can be implemented in an image gallery desktop application to allow efficient browsing through three different search modes: retrieval by image query, retrieval by facial recognition, and retrieval by text or tags. In this project, the MPEG-7-like Powered Localized Color and Edge Directivity Descriptor is used to extract the feature vectors of the image database and the facial recognition system is built around the Eigenfaces concept. A graphical user interface with the basic functionality of an image gallery application is also developed to implement the three search modes. Results show that the application is able to retrieve and display images in a collection as thumbnail previews with high retrieval accuracy and medium relevance and the computational requirements for subsequent searches were significantly reduced through the incorporation of text-based image retrieval as one of the search modes. All in all, this study introduces a simple and convenient way of offline image searches on desktop computers and provides a stepping stone to future content-based image retrieval systems built for similar purposes

Image retrieval : a first step for a human centered approach

International audienceImage indexing using content analysis is known as a difficult task, involving the vision research domain. Using these tools in the context of a retrieval system is generally frustrating for users, due to a lack of interfaces development, and to the difficulty for users to understand the low-level features managed by the system. We propose in this paper a general point of view for introducing a link between such systems and potential users. This includes image features based on visual perception models, a relevance feedback model, and a graphical interface to express the information need through user-system interaction

A framework for understanding user interaction with content-based image retrieval: model, interface and users

User interaction is essential to the communication between users and content-based image retrieval (CBIR) systems. User interaction covers three key elements: an interaction model, an interactive interface and users. The three key elements combine to enable effective interaction to happen. Many studies have investigated different aspects of user interaction. However, there is lack of research in combining all three elements in an integrated manner, especially through well-principled data analysis based on a systematic user study. In this thesis, we investigate the combination of all three elements for interactive CBIR. We first propose uInteract - a framework including a novel four-factor user interaction model (FFUIM) and an interactive interface. The FFUIM aims to improve interaction and search accuracy of the relevance feedback mechanism for CBIR. The interface delivers the FFUIM visually, aiming to support users in grasping how the interaction model functions and how best to manipulate it. The framework is tested in three task-based and user-oriented comparative evaluations, which involves 12 comparative systems, 12 real life scenario tasks and 50 subjects. The quantitative data analysis shows encouraging observations on ease of use and usefulness of the proposed framework, and also reveals a large variance of the results depending on different user types. Accordingly, based on Information Foraging Theory, we further propose a user classification model along three user interaction dimensions: information goals (I), search strategies (S) and evaluation thresholds (E) of users. To our best knowledge, this is the first principled user classification model in CBIR. The model is operated and verified by a systematic qualitative data analysis based on multi linear regression on the real user interaction data from comparative user evaluations. From final quantitative and qualitative data analysis based on the ISE model, we have established what different types of users like about the framework and their preferences for interactive CBIR systems. Our findings offer useful guidelines for interactive search system design, evaluation and analysis

Video browsing interfaces and applications: a review

We present a comprehensive review of the state of the art in video browsing and retrieval systems, with special emphasis on interfaces and applications. There has been a significant increase in activity (e.g., storage, retrieval, and sharing) employing video data in the past decade, both for personal and professional use. The ever-growing amount of video content available for human consumption and the inherent characteristics of video data—which, if presented in its raw format, is rather unwieldy and costly—have become driving forces for the development of more effective solutions to present video contents and allow rich user interaction. As a result, there are many contemporary research efforts toward developing better video browsing solutions, which we summarize. We review more than 40 different video browsing and retrieval interfaces and classify them into three groups: applications that use video-player-like interaction, video retrieval applications, and browsing solutions based on video surrogates. For each category, we present a summary of existing work, highlight the technical aspects of each solution, and compare them against each other

Towards an All-Purpose Content-Based Multimedia Information Retrieval System

The growth of multimedia collections - in terms of size, heterogeneity, and variety of media types - necessitates systems that are able to conjointly deal with several forms of media, especially when it comes to searching for particular objects. However, existing retrieval systems are organized in silos and treat different media types separately. As a consequence, retrieval across media types is either not supported at all or subject to major limitations. In this paper, we present vitrivr, a content-based multimedia information retrieval stack. As opposed to the keyword search approach implemented by most media management systems, vitrivr makes direct use of the object's content to facilitate different types of similarity search, such as Query-by-Example or Query-by-Sketch, for and, most importantly, across different media types - namely, images, audio, videos, and 3D models. Furthermore, we introduce a new web-based user interface that enables easy-to-use, multimodal retrieval from and browsing in mixed media collections. The effectiveness of vitrivr is shown on the basis of a user study that involves different query and media types. To the best of our knowledge, the full vitrivr stack is unique in that it is the first multimedia retrieval system that seamlessly integrates support for four different types of media. As such, it paves the way towards an all-purpose, content-based multimedia information retrieval system

An adaptive technique for content-based image retrieval

We discuss an adaptive approach towards Content-Based Image Retrieval. It is based on the Ostensive Model of developing information needs—a special kind of relevance feedback model that learns from implicit user feedback and adds a temporal notion to relevance. The ostensive approach supports content-assisted browsing through visualising the interaction by adding user-selected images to a browsing path, which ends with a set of system recommendations. The suggestions are based on an adaptive query learning scheme, in which the query is learnt from previously selected images. Our approach is an adaptation of the original Ostensive Model based on textual features only, to include content-based features to characterise images. In the proposed scheme textual and colour features are combined using the Dempster-Shafer theory of evidence combination. Results from a user-centred, work-task oriented evaluation show that the ostensive interface is preferred over a traditional interface with manual query facilities. This is due to its ability to adapt to the user's need, its intuitiveness and the fluid way in which it operates. Studying and comparing the nature of the underlying information need, it emerges that our approach elicits changes in the user's need based on the interaction, and is successful in adapting the retrieval to match the changes. In addition, a preliminary study of the retrieval performance of the ostensive relevance feedback scheme shows that it can outperform a standard relevance feedback strategy in terms of image recall in category search

An Investigation on Text-Based Cross-Language Picture Retrieval Effectiveness through the Analysis of User Queries

Purpose: This paper describes a study of the queries generated from a user experiment for cross-language information retrieval (CLIR) from a historic image archive. Italian speaking users generated 618 queries for a set of known-item search tasks. The queries generated by user’s interaction with the system have been analysed and the results used to suggest recommendations for the future development of cross-language retrieval systems for digital image libraries. Methodology: A controlled lab-based user study was carried out using a prototype Italian-English image retrieval system. Participants were asked to carry out searches for 16 images provided to them, a known-item search task. User’s interactions with the system were recorded and queries were analysed manually quantitatively and qualitatively. Findings: Results highlight the diversity in requests for similar visual content and the weaknesses of Machine Translation for query translation. Through the manual translation of queries we show the benefits of using high-quality translation resources. The results show the individual characteristics of user’s whilst performing known-item searches and the overlap obtained between query terms and structured image captions, highlighting the use of user’s search terms for objects within the foreground of an image. Limitations and Implications: This research looks in-depth into one case of interaction and one image repository. Despite this limitation, the discussed results are likely to be valid across other languages and image repository. Value: The growing quantity of digital visual material in digital libraries offers the potential to apply techniques from CLIR to provide cross-language information access services. However, to develop effective systems requires studying user’s search behaviours, particularly in digital image libraries. The value of this paper is in the provision of empirical evidence to support recommendations for effective cross-language image retrieval system design.</p

An explorative study of interface support for image searching

In this paper we study interfaces for image retrieval systems. Current image retrieval interfaces are limited to providing query facilities and result presentation. The user can inspect the results and possibly provide feedback on their relevance for the current query. Our approach, in contrast, encourages the user to group and organise their search results and thus provide more fine-grained feedback for the system. It combines the search and management process, which - according to our hypothesis - helps the user to onceptualise their search tasks and to overcome the query formulation problem. An evaluation, involving young design-professionals and di®erent types of information seeking scenarios, shows that the proposed approach succeeds in encouraging the user to conceptualise their tasks and that it leads to increased user satisfaction. However, it could not be shown to increase performance. We identify the problems in the current setup, which when eliminated should lead to more effective searching overall