Unsupervised Visual and Textual Information Fusion in Multimedia Retrieval - A Graph-based Point of View

Multimedia collections are more than ever growing in size and diversity. Effective multimedia retrieval systems are thus critical to access these datasets from the end-user perspective and in a scalable way. We are interested in repositories of image/text multimedia objects and we study multimodal information fusion techniques in the context of content based multimedia information retrieval. We focus on graph based methods which have proven to provide state-of-the-art performances. We particularly examine two of such methods : cross-media similarities and random walk based scores. From a theoretical viewpoint, we propose a unifying graph based framework which encompasses the two aforementioned approaches. Our proposal allows us to highlight the core features one should consider when using a graph based technique for the combination of visual and textual information. We compare cross-media and random walk based results using three different real-world datasets. From a practical standpoint, our extended empirical analysis allow us to provide insights and guidelines about the use of graph based methods for multimodal information fusion in content based multimedia information retrieval.Comment: An extended version of the paper: Visual and Textual Information Fusion in Multimedia Retrieval using Semantic Filtering and Graph based Methods, by J. Ah-Pine, G. Csurka and S. Clinchant, submitted to ACM Transactions on Information System

The Wikipedia Image Retrieval Task

The wikipedia image retrieval task at ImageCLEF provides a testbed for the system-oriented evaluation of visual information retrieval from a collection of Wikipedia images. The aim is to investigate the effectiveness of retrieval approaches that exploit textual and visual evidence in the context of a large and heterogeneous collection of images that are searched for by users with diverse information needs. This chapter presents an overview of the available test collections, summarises the retrieval approaches employed by the groups that participated in the task during the 2008 and 2009 ImageCLEF campaigns, provides an analysis of the main evaluation results, identifies best practices for effective retrieval, and discusses open issues

AMBIT: Semantic Engine Foundations for Knowledge Management in Context-dependent Applications

Context-aware application and services proposing potentially useful information to users are more and more widespread; however, their actual usefulness is often limited by the “syntactical” notion of context they adopt. The recently started AMBIT project aims to provide a general software architecture for developing semantic-based context-aware tools in a number of vertical case study applications. In this paper, we focus on the knowledge management foundations we are laying for the Semantic Engine of the AMBIT architecture. The proposed semantic analysis and similarity techniques: (a) exploit the textual information deeply characterizing both users and the information to be retrieved; (b) overcome the limits of syntactic methods by leveraging on the strengths of both classic information retrieval and knowledge-based analysis and classification, ultimately proposing information relevant to the user interests. The experimental evaluation of a preliminary implementation in an actual “cultural territorial enhancement” scenario already shows promising results

A study of search intermediary working notes: implications for IR system design

This paper reports findings from an exploratory study investigating working notes created during encoding and external storage (EES) processes, by human search intermediates using a Boolean information retrieval (JR) system. EES processes have been an important area of research in educational contexts where students create and use notes to facilitate learning. In the context of interactive IR, encoding can be conceptualized as the process of creating working notes to help in the understanding and translating a user's information problem into a search strategy suitable for use with an IR system. External storage is the process of using working notes to facilitate interaction with IR systems. Analysis of 221 sets of working notes created by human search intermediaries revealed extensive use of EES processes and the creation of working notes of textual, numerical and graphical entities. Nearly 70% of recorded working notes were textual/numerical entities, nearly 30% were graphical entities and 0.73% were indiscernible. Segmentation devices were also used in 48% of the working notes. The creation of working notes during EES processes was a fundamental element within the mediated, interactive IR process. Implications for the design of IR interfaces to support users' EES processes and further research is discussed

The Wikipedia Image Retrieval Task

htmlabstractThe wikipedia image retrieval task at ImageCLEF provides a testbed for the system-oriented evaluation of visual information retrieval from a collection of Wikipedia images. The aim is to investigate the effectiveness of retrieval approaches that exploit textual and visual evidence in the context of a large and heterogeneous collection of images that are searched for by users with diverse information needs. This chapter presents an overview of the available test collections, summarises the retrieval approaches employed by the groups that participated in the task during the 2008 and 2009 ImageCLEF campaigns, provides an analysis of the main evaluation results, identifies best practices for effective retrieval, and discusses open issues

Exploring Supervised Techniques for Automated Recognition of Intention Classes from Portuguese Free Texts on Agriculture

Technical and scientific knowledge is vast and complex, particularly in interdisciplinary fields such as sustainable agriculture, which is available in several interrelated, geographically dispersed and interdisciplinary online textual information sources. In this context, it is essential to support people with computational mechanisms that allow them to retrieve and interpret information in an appropriate way, as communication in these software systems is typically asynchronous and textual. User’s intention recognition and analysis in textual documents results in benefits for better information retrieval. However, intentions are expressed implicitly in texts in natural language and the specificities of the domain and cultural aspects of language make it difficult to process and analyze the text by computer systems. This requires the study of methods for the automatic recognition of intention classes in text. In this article, we conduct extensive experimental analyses on techniques based on language models and machine learning to detect instances of intention classes in texts about sustainable agriculture written in Portuguese. In our methodology, we perform a morphological analysis of the sentences and evaluate four Word Embeddings techniques (Word2Vec, Wang2Vec, FastText and Glove) combined with four machine learning techniques (Support Vector Machine, Artificial Neural Network, Random Forest and Transfer Learning). The results obtained by applying the techniques proposed in a database with textual information on sustainable agriculture indicate promising possibilities in the recognition of intentions in free texts  in  Portuguese language on sustainable agriculture

Multimedia search without visual analysis: the value of linguistic and contextual information

This paper addresses the focus of this special issue by analyzing the potential contribution of linguistic content and other non-image aspects to the processing of audiovisual data. It summarizes the various ways in which linguistic content analysis contributes to enhancing the semantic annotation of multimedia content, and, as a consequence, to improving the effectiveness of conceptual media access tools. A number of techniques are presented, including the time-alignment of textual resources, audio and speech processing, content reduction and reasoning tools, and the exploitation of surface features