Improving instance search performance in video collections

This thesis presents methods to improve instance search and enhance user performance while browsing unstructured video collections. Through the use of computer vision and information retrieval techniques, we propose novel solutions to analyse visual content and build a search algorithm to address the challenges of visual instance search, while considering the constraints for practical applications. Firstly, we investigate methods to improve the effectiveness of instance search systems for finding object instances which occurred in unstructured video content. Using the bag of feature framework, we propose a novel algorithm to use the geometric correlation information between local features to improve the accuracy of local feature matching, thus improve the performance of instance search systems without introducing much computation cost. Secondly, we consider the scenario that the performance of instance search systems may drop due to the volume of visual content in large video collections. We introduce a search algorithm based on embedded coding to increase the effectiveness and efficiency of instance search systems. And we participate in the international video evaluation campaign, TREC Video Retrieval Evaluation, to comparatively evaluate the performance of our proposed methods. Finally, the exploration and navigation of visual content when browsing large unstructured video collections is considered. We propose methods to address such challenges and build an interactive video browsing tool to improve user performance while seeking interesting content over video collections. We construct a structured content representation with similarity graph using our proposed instance search technologies. Considering the constraints related to real world usability, we present a flexible interface based on faceted navigation to enhance user performance when completing video browsing tasks. This thesis shows that user performance can be enhanced by improving the effectiveness of instance search approaches, when seeking information in unstructured video collection. While covering many different aspects of improving instance search in this work, we outline three potential directions for future work: advanced feature representation, data driven rank and cloud-based search algorithms

Digital Image Access & Retrieval

The 33th Annual Clinic on Library Applications of Data Processing, held at the University of Illinois at Urbana-Champaign in March of 1996, addressed the theme of "Digital Image Access & Retrieval." The papers from this conference cover a wide range of topics concerning digital imaging technology for visual resource collections. Papers covered three general areas: (1) systems, planning, and implementation; (2) automatic and semi-automatic indexing; and (3) preservation with the bulk of the conference focusing on indexing and retrieval.published or submitted for publicatio

Accurator: Nichesourcing for Cultural Heritage

With more and more cultural heritage data being published online, their usefulness in this open context depends on the quality and diversity of descriptive metadata for collection objects. In many cases, existing metadata is not adequate for a variety of retrieval and research tasks and more specific annotations are necessary. However, eliciting such annotations is a challenge since it often requires domain-specific knowledge. Where crowdsourcing can be successfully used for eliciting simple annotations, identifying people with the required expertise might prove troublesome for tasks requiring more complex or domain-specific knowledge. Nichesourcing addresses this problem, by tapping into the expert knowledge available in niche communities. This paper presents Accurator, a methodology for conducting nichesourcing campaigns for cultural heritage institutions, by addressing communities, organizing events and tailoring a web-based annotation tool to a domain of choice. The contribution of this paper is threefold: 1) a nichesourcing methodology, 2) an annotation tool for experts and 3) validation of the methodology and tool in three case studies. The three domains of the case studies are birds on art, bible prints and fashion images. We compare the quality and quantity of obtained annotations in the three case studies, showing that the nichesourcing methodology in combination with the image annotation tool can be used to collect high quality annotations in a variety of domains and annotation tasks. A user evaluation indicates the tool is suited and usable for domain specific annotation tasks

Visual object category discovery in images and videos

textThe current trend in visual recognition research is to place a strict division between the supervised and unsupervised learning paradigms, which is problematic for two main reasons. On the one hand, supervised methods require training data for each and every category that the system learns; training data may not always be available and is expensive to obtain. On the other hand, unsupervised methods must determine the optimal visual cues and distance metrics that distinguish one category from another to group images into semantically meaningful categories; however, for unlabeled data, these are unknown a priori. I propose a visual category discovery framework that transcends the two paradigms and learns accurate models with few labeled exemplars. The main insight is to automatically focus on the prevalent objects in images and videos, and learn models from them for category grouping, segmentation, and summarization. To implement this idea, I first present a context-aware category discovery framework that discovers novel categories by leveraging context from previously learned categories. I devise a novel object-graph descriptor to model the interaction between a set of known categories and the unknown to-be-discovered categories, and group regions that have similar appearance and similar object-graphs. I then present a collective segmentation framework that simultaneously discovers the segmentations and groupings of objects by leveraging the shared patterns in the unlabeled image collection. It discovers an ensemble of representative instances for each unknown category, and builds top-down models from them to refine the segmentation of the remaining instances. Finally, building on these techniques, I show how to produce compact visual summaries for first-person egocentric videos that focus on the important people and objects. The system leverages novel egocentric and high-level saliency features to predict important regions in the video, and produces a concise visual summary that is driven by those regions. I compare against existing state-of-the-art methods for category discovery and segmentation on several challenging benchmark datasets. I demonstrate that we can discover visual concepts more accurately by focusing on the prevalent objects in images and videos, and show clear advantages of departing from the status quo division between the supervised and unsupervised learning paradigms. The main impact of my thesis is that it lays the groundwork for building large-scale visual discovery systems that can automatically discover visual concepts with minimal human supervision.Electrical and Computer Engineerin

Beyond Search

This research delves into the realm of digital audiovisual (AV) archives, focusing on user experience and advocating for the integration of exploratory approaches alongside conventional search. Within cultural heritage institutions, conventional keyword-based search interfaces have long served as the primary means to access digital AV archives. However, these interfaces often fall short in addressing the diverse needs of users and serving more exploratory or open-ended queries. Drawing on a series of illustrative case studies, this report showcases innovative practices in the cultural heritage domain. Furthermore, it looks beyond archives to seek inspiration from practitioners in other disciplines, such as artists, filmmakers, and community initiatives grappling with similar questions. The research report identifies four core themes: Generous + Fluid Interfaces; Situated + Experiential Entry Points; Computational Sensing + Algorithmic Metadata; and Participatory Sense-Making + Storytelling. Each theme offers distinct benefits in terms of user engagement, accessibility, contextualization, and storytelling. Challenges of complexity, accessibility, and compatibility are also discussed. This research endeavors to redefine the potential of the interaction paradigm and offer a rich set of pathways, where digital AV archives transcend conventional search methods to offer immersive, dynamic, user-centric experiences. By integrating exploratory interfaces, cultural heritage institutions can unlock the full potential of their collections, making them more engaging and accessible to a broader audience