Low Level Processing of Audio and Video Information for Extracting the Semantics of Content

The problem of semantic indexing of multimedia documents is actually of great interest due to the wide diffusion of large audio-video databases. We first briefly describe some techniques used to extract low-level features (e.g., shot change detection, dominant color extraction, audio classification etc.). Then the ToCAI (table of contents and analytical index) framework for content description of multimedia material is presented, together with an application which implements it. Finally we propose two algorithms suitable for extracting the high level semantics of a multimedia document. The first is based on finite-state machines and low-level motion indices, whereas the second uses hidden Markov models

A Video Indexing Approach Based on Audio Classification

This paper presents a video indexing approach based only on audio classification. Indeed, we apply to an audio-visual document a set of methods for partitioning the associated audio data into homogeneous segments. The aim is to highlight semantically relevant items of a multimedia document by relying only on simple audio processing techniques. A simple algorithm to identify audio segments belonging to silence, music, speech and noise classes has been proposed

Identifying Video Content Consistency by Vector Quantization

Many post-production videos such as movies and cartoons present well structured story-lines organized in separated visual scenes. Accurate grouping of shots into these logical segments could lead to semantic indexing of scenes for interactive multimedia retrieval and video summaries. In this paper we introduce a novel shot-based analysis approach which aims to cluster together shots with similar visual content. We demonstrate how the use of codebooks of visual codewords (generated by a vector quantization process) represents an effective method to identify clusters containing shots with similar long-term consistency of chromatic compositions. The clusters, obtained by a single-link clustering algorithm, allow the further use of the well-known scene transition graph framework for logical story unit detection and pattern investigation

Multimedia documents description by ordered hierarchies: the ToCAIdescription scheme

The authors present the ToCAI (Table of Content Analytical Index) framework, a description scheme (DS) for content description of audio-visual (AV) documents. The idea for such a description scheme comes from the structures used for indexing technical books (table of content and analytical index). This description scheme provides therefore a hierarchical description of the time sequential structure of a multimedia document (ToC), suitable for browsing, together with an “Analytical Index” (AI) of the key items of the document, suitable for retrieval. The AI allows one to represent in a ordered way the items of the AV document which are most relevant from the semantic point of view. The ordering criteria are therefore selected according to the application context. The detailed structure of the DS is presented by means of UML notation and an application example is also shown

Describing multimedia documents in natural and semantic-driven ordered hierarchies

In this work we present the ToCAI (Table of Content-Analytical Index) framework, a description scheme (DS) for content description of audio-visual (AV) documents. The idea for such a description scheme comes out from the structures used for indexing technical books (table of content and analytical index). This description scheme provides therefore a hierarchical description of the time sequential structure of a multimedia document (ToC), suitable for browsing, together with an analytical index (AI) of the key items of the document, suitable for retrieval. The AI allows to represent in an ordered way the items of the AV document which are most relevant from the semantic point of view. The ordering criteria are therefore selected according to the application context. The detailed structure of the DS is presented by means of UML notation as well and an application example is shown

Audio-Visual Pattern Recognition using HMM for Content-Based Multimedia Indexing

The aim of this work consists in the development of automatic techniques for the extraction of content-based information from audiovisual data. The focus has been placed on providing tools for analyzing both audio and visual streams, for translating the signal samples into sequences of indices. The signal classification are performed by means of Hidden Markov Models (HMM), used in an innovative approach: the input signal is considered as a non-stationary stochastic process, modeled by a HMM in which each state stands for a different class of the signal. This defines an adaptive classification scheme for which a set of new training algorithms has been developed. Several samples from the MPEG-7 content set have been analyzed using the proposed classification scheme, demonstrating the performance of the overall approach to provide insights of the content of the audio-visual material

ToCAI: A Framework for Indexing and Retrieval of Multimedia Documents

This paper presents the ToCAI (table of content-analytical index) description scheme (DS) for content description of audio-visual documents. The original idea comes from the structure used for technical books. One may easily understand a book's sequential organization by looking at its table of contents while quickly retrieving elements of interest by means of the analytical index. This description scheme provides therefore a hierarchical description of the time sequential structure of a multimedia document (thanks to the ToC), suitable for browsing, together with an “analytical index” (AI) of audio-visual objects of the document, suitable for effective retrieval. Besides, two sub-description schemes for information about description generation and about the metadata associated with the document are also enclosed in the general DS. The detailed structure of the DS is also presented by means of UML (unified modelling language) notation and an application example is shown. Finally, some considerations concerning the adopted visual interface are made

Indexing Audio-Visual Sequences by Joint Audio and Video Processing

The focus of this work is oriented to the creation of a content-based hierarchical organisation of audio-visual data (a description scheme) and to the creation of meta-data (descriptors) to associate with audio and/or visual signals. The generation of efficient indices to access audio-visual databases is strictly connected to the generation of content descriptors and to the hierarchical representation of audio-visual material. Once a hierarchy can be extracted from the data analysis, a nested indexing structure can be created to access relevant information at a specific level of detail. Accordingly, a query can be made very specific in relationship to the level of detail that is required by the user. In order to construct the hierarchy, we describe how to extract information content from audio-visual sequences so as to have different hierarchical indicators (or descriptors), which can be associated to each media (audio, video). At this stage, video and audio signals can be separated into temporally consistent elements. At the lowest level, information is organised in frames (groups of pixels for visual information, groups of consecutive samples for audio information). At a higher level, low-level consistent temporal entities are identified: in case of digital image sequences, these consist of shots (or continuous camera records) which can be obtained by detecting cuts or special effects such as dissolves, fade in and fade out; in case of audio information, these represent consistent audio segments belonging to one specific audio type (such as speech, music, silence, ...). One more level up, patterns of video shots or audio segments can be recognised so as to reflect more meaningful structures such as dialogues, actions, ... At the highest level, information is organised so as to establish correlation beyond the temporal organisation of information, allowing to reflect classes of visual or audio types: we call these classes idioms. The paper ends with a description of possible solutions to allow a cross-modal analysis of audio and video information, which may validate or invalidate the proposed hierarchy, and in some cases enable more sophisticated levels of representation of information content

Audio-Visual VQ Shot Clustering for Video Programs

Many post-production video documents such as movies, sitcoms and cartoons present well structured story-lines organized in separated audio-visual scenes. Accurate grouping of shots into these logical video segments could lead to semantic indexing of scenes and events for interactive multimedia retrieval. In this paper we introduce a novel shot based analysis approach which aims to cluster together shots with similar audio-visual content. We demonstrate how the use of codebooks of audio and visual codewords (generated by a vector quantization process) results to be an effective method to represent clusters containing shots with similar long-term consistency of chromatic compositions and audio. The output clusters obtained by a simple single-link clustering algorithm, allow the further application of the well-known scene transition graph framework for scene change detection and shot-pattern investigation. In the end the merging of audio and visual results leads to a hierarchical description of the whole video document, useful for multimedia retrieval and summarization purposes

Scene extraction in motion pictures

This paper addresses the challenge of bridging the semantic gap between the rich meaning users desire when they query to locate and browse media and the shallowness of media descriptions that can be computed in today\u27s content management systems. To facilitate high-level semantics-based content annotation and interpretation, we tackle the problem of automatic decomposition of motion pictures into meaningful story units, namely scenes. Since a scene is a complicated and subjective concept, we first propose guidelines from fill production to determine when a scene change occurs. We then investigate different rules and conventions followed as part of Fill Grammar that would guide and shape an algorithmic solution for determining a scene. Two different techniques using intershot analysis are proposed as solutions in this paper. In addition, we present different refinement mechanisms, such as film-punctuation detection founded on Film Grammar, to further improve the results. These refinement techniques demonstrate significant improvements in overall performance. Furthermore, we analyze errors in the context of film-production techniques, which offer useful insights into the limitations of our method