CHORUS Deliverable 2.1: State of the Art on Multimedia Search Engines

Based on the information provided by European projects and national initiatives related to multimedia search as well as domains experts that participated in the CHORUS Think-thanks and workshops, this document reports on the state of the art related to multimedia content search from, a technical, and socio-economic perspective. The technical perspective includes an up to date view on content based indexing and retrieval technologies, multimedia search in the context of mobile devices and peer-to-peer networks, and an overview of current evaluation and benchmark inititiatives to measure the performance of multimedia search engines. From a socio-economic perspective we inventorize the impact and legal consequences of these technical advances and point out future directions of research

Digital Image Access & Retrieval

The 33th Annual Clinic on Library Applications of Data Processing, held at the University of Illinois at Urbana-Champaign in March of 1996, addressed the theme of "Digital Image Access & Retrieval." The papers from this conference cover a wide range of topics concerning digital imaging technology for visual resource collections. Papers covered three general areas: (1) systems, planning, and implementation; (2) automatic and semi-automatic indexing; and (3) preservation with the bulk of the conference focusing on indexing and retrieval.published or submitted for publicatio

A Video Library System Using Scene Detection and Automatic Tagging

We present a novel video browsing and retrieval system for edited videos, in which videos are automatically decomposed into meaningful and storytelling parts (i.e. scenes) and tagged according to their transcript. The system relies on a Triplet Deep Neural Network which exploits multimodal features, and has been implemented as a set of extensions to the eXo Platform Enterprise Content Management System (ECMS). This set of extensions enable the interactive visualization of a video, its automatic and semi-automatic annotation, as well as a keyword-based search inside the video collection. The platform also allows a natural integration with third-party add-ons, so that automatic annotations can be exploited outside the proposed platform

Affective Recommendation of Movies Based on Selected Connotative Features

The apparent difficulty in assessing emotions elicited by movies and the undeniable high variability in subjects emotional responses to filmic content have been recently tackled by exploring film connotative properties: the set of shooting and editing conventions that help in transmitting meaning to the audience. Connotation provides an intermediate representation which exploits the objectivity of audiovisual descriptors to predict the subjective emotional reaction of single users. This is done without the need of registering users physiological signals neither by employing other people highly variable emotional rates, but just relying on the inter-subjectivity of connotative concepts and on the knowledge of users reactions to similar stimuli. This work extends previous by extracting audiovisual and film grammar descriptors and, driven by users rates on connotative properties, creates a shared framework where movie scenes are placed, compared and recommended according to connotation. We evaluate the potential of the proposed system by asking users to assess the ability of connotation in suggesting filmic content able to target their affective requests

Making Public Media Personal: Nostalgia and Reminiscence in the Office

In this paper we explore the notion of creating personally evocative collections of content from publicly available material. Compared to the personal media that we look at, reminisce over, or personalise our offices with, public media offers the potential for a different type of nostalgia, signifiers of an era such as entertainment, products, or fashions. We focus on an office environment, where the use of filtered public media may mitigate concerns over protecting privacy and disclosing too much of one's identity, while keeping the existing benefits of office personalisation in terms of reminiscence, improving mood, and developing identity. After preliminary explorations of content and form, we developed a two-screen ambient display that cycled through 500 images automatically retrieved based on four simple user questions. We ran a two-week trial of the display with six users. We present qualitative results of the trial from which we see that it is possible to bring the delight associated with personal content into the workplace, while being mindful of issues of appropriateness and privacy. Images of locations from childhood were particularly evocative for all participants, while simple objects such as stickers, music, or boardgames were more varied across participants. We discuss a number of avenues for future work in the workplace and beyond: improving the chance of an evocative moment, capturing the mundane, and the crowdsourcing of nostalgia

Event Based Retrieval From Digital Libraries Containing Data Streams

The objective of this research is to study the issues involved in building a digital library that contains data streams and allows event-based retrieval. “Digital Libraries are storehouses of information available through the Internet that provide ways to collect, store, and organize data and make it accessible for search, retrieval, and processing” [29]. Data streams are sources of information for applications such as news-on-demand, weather services, and scientific research, to name a few. A data stream is a sequence of data units produced over a period of time. Examples of data streams are video streams, audio stream, and sensor readings. Saving data streams in digital libraries is advantageous because of the services provided by digital libraries such as archiving, preservation, administration, and access control. Events are noteworthy occurrences that happen during data streams. Events are easier to remember than specific time instances at which they occur; hence using them for retrieval is more commensurate with human behavior and can be more efficient via direct accessing instead of scanning. The focus of this research is not only on storing data streams in a digital library and using event-based retrieval, but also on relating streams and playing them back at the same time, possibly in a synchronized manner, to facilitate better understanding in research or other working situations. Our approach for this research starts by considering digital libraries for: stock market, news streams, census bureau statistics, weather, sports games, and the educational environment. For each of these applications, we form categories of possible users and the basic requirements for each of them. As a result, we identify a list of design goals that we take into consideration in developing the architecture of the library. To illustrate and validate our approach we implement a medical digital library containing actual Computed Tomography (CT) scan streams. It also contains sample medical text and audio streams to show the heterogeneity of the library. Streams are displayed in a concise, yet complete, way that makes it unproblematic for users to decide whether or not to playback a stream and to set playback options. The playback interface itself is organized in a way that accommodates synchronous and asynchronous streams and enables users to control the playback of these streams. We study the performance of the specialized search and retrieval processes in comparison to traditional search and retrieval processes. We conclude with a discussion on how to adapt the library to additional stream types in addition to suggesting other future efforts in this area

An audio-visual approach to web video categorization

International audienceIn this paper we address the issue of automatic video genre categorization of web media using an audio-visual approach. To this end, we propose content descriptors which exploit audio, temporal structure and color information. The potential of our descriptors is experimentally validated both from the perspective of a classification system and as an information retrieval approach. Validation is carried out on a real scenario, namely on more than 288 hours of video footage and 26 video genres specific to blip.tv media platform. Additionally, to reduce semantic gap, we propose a new relevance feedback technique which is based on hierarchical clustering. Experimental tests prove that retrieval performance can be significantly increased in this case, becoming comparable to the one obtained with high level semantic textual descriptors

Exploring film language with a digital analysis tool: The case of kinolab

This article presents a case study of Kinolab, a digital platform for the analysis of narrative film language. It describes the need for a scholarly database of clips focusing on film language for cinema and media studies faculty and students, highlighting recent technological and legal advances that have created a favorable environment for this kind of digital humanities work. Discussion of the project is situated within the broader context of contemporary developments in moving image annotation and a discussion of the unique challenges posed by computationally-driven moving image analysis. The article also argues for a universally accepted data model for film language to facilitate the academic crowdsourcing of film clips and the sharing of research and resources across the Semantic Web

Semantic browsing of digital collections

Visiting museums is an increasingly popular pastime. Studies have shown that visitors can draw on their museum experience, long after their visit, to learn new things in practical situations. Rather than viewing a visit as a single learning event, we are interested in ways of extending the experience to allow visitors to access online resources tailored to their interests. Museums typically have extensive archives that can be made available online, the challenge is to match these resources to the visitor’s interests and present them in a manner that facilitates exploration and engages the visitor. We propose the use of knowledge level resource descriptions to identify relevant resources and create structured presentations. A system that embodies this approach, which is in use in a UK museum, is presented and the applicability of the approach to the broader semantic web is discussed

Random Assisted Browsing of Rushes Archives

How to efficiently browse a large video database if its content is unknown to the user? In this paper we propose new approaches for browsing initialisation, exploration and content access of a rushes archive, where the span of information stored can be huge and difficult to understand at a glance. Exploring and navigating through raw footage is assisted by organising the video material in a meaningful structure and by adopting appropriate visualisation solutions. Un-annotated content is organised in hierarchical previews, while browsing is enabled by novel methods of random exploration and random content access to preview nodes. User tests conducted on professional users in a real-work scenario aim at demonstrating how the hierarchical visualisation and the proposed random browsing solutions assist the process of accessing and retrieving desired content