Clustering-based analysis of semantic concept models for video shots

In this paper we present a clustering-based method for representing semantic concepts on multimodal low-level feature spaces and study the evaluation of the goodness of such models with entropy-based methods. As different semantic concepts in video are most accurately represented with different features and modalities, we utilize the relative model-wise confidence values of the feature extraction techniques in weighting them automatically. The method also provides a natural way of measuring the similarity of different concepts in a multimedia lexicon. The experiments of the paper are conducted using the development set of the TRECVID 2005 corpus together with a common annotation for 39 semantic concept

Measuring concept similarities in multimedia ontologies: analysis and evaluations

The recent development of large-scale multimedia concept ontologies has provided a new momentum for research in the semantic analysis of multimedia repositories. Different methods for generic concept detection have been extensively studied, but the question of how to exploit the structure of a multimedia ontology and existing inter-concept relations has not received similar attention. In this paper, we present a clustering-based method for modeling semantic concepts on low-level feature spaces and study the evaluation of the quality of such models with entropy-based methods. We cover a variety of methods for assessing the similarity of different concepts in a multimedia ontology. We study three ontologies and apply the proposed techniques in experiments involving the visual and semantic similarities, manual annotation of video, and concept detection. The results show that modeling inter-concept relations can provide a promising resource for many different application areas in semantic multimedia processing

Indexing and retrieval of free-form surfaces using self-organizing maps

This thesis describes the use of Self-Organizing Maps in combination with unique curvature based feature combinations and various training data sets as a clustering function to describe free-form surfaces. Our descriptor was successfully used as the basis of techniques that segmented, indexed and retrieved free-form surfaces represented as triangulated meshes

Innehållsbaserad sökning av hierarkiska objekt med PicSOM

The amounts of multimedia content available to the public have been increasing rapidly in the last decades and it is expected to grow exponentially in the years to come. This development puts an increasing emphasis on automated content-based information retrieval (CBIR) methods, which index and retrieve multimedia based on its contents. Such methods can automatically process huge amounts of data without the human intervention required by traditional methods (e.g. manual categorisation, entering of keywords). Unfortunately CBIR methods do have a serious problem: the so-called semantic gap between the low-level descriptions used by computer systems and the high-level concepts of humans. However, by emulating human skills such as understanding the contexts and relationships of the multimedia objects one might be able to bridge the semantic gap. To this end, this thesis proposes a method of using hierarchical objects combined with relevance sharing. The proposed method can incorporate natural relationships between multimedia objects and take advantage of these in the retrieval process, hopefully improving the retrieval accuracy considerably. The literature survey part of the thesis consists of a review of content-based information retrieval in general and also looks at multimodal fusion in CBIR systems and how that has been implemented previously in different scenarios. The work performed for this thesis includes the implementation of hierarchical objects and multimodal relevance sharing into the PicSOM CBIR system. Also extensive experiments with different kinds of multimedia and other hierarchical objects (segmented images, web-link structures and video retrieval) were performed to evaluate the usefulness of the hierarchical objects paradigm. Keywords: content-based retrieval, self-organizing map, multimedia database

Video Summarization with SOMs

Video summarization is a process where a long video file is converted to a considerably shorter form. The video summary can then be used to facilitate efficient searching and browsing of video files in large video collections. The aim of successful automatic summarization is to preserve as much as possible from the essential content of each video. What is essential is of course subjective and also dependent on the use of the videos and the overall content of the collection. In this paper we present an overview of the SOM-based methodology we have used for video summarization, which analyzes the temporal trajectories of the best-matching units of frame-wise feature vectors. It has been developed as a part of PicSOM, our content-based multimedia information retrieval and analysis framework. The video material we have used in our experiments comes from NIST's annual TRECVID evaluation for content-based video retrieval systems