The problem of semantic indexing of multimedia documents is actually of great interest due to the wide diffusion of large audio-video databases. We first briefly describe some techniques used to extract low-level features (e.g., shot change detection, dominant color extraction, audio classification etc.). Then the ToCAI (table of contents and analytical index) framework for content description of multimedia material is presented, together with an application which implements it. Finally we propose two algorithms suitable for extracting the high level semantics of a multimedia document. The first is based on finite-state machines and low-level motion indices, whereas the second uses hidden Markov models

BUGATTI A

LEONARDI R

MIGLIORATI P

N. ADAMI

ADAMI, Nicola

BUGATTI, Alessandro

LEONARDI, Riccardo

MIGLIORATI, Pierangelo

Archivio istituzionale della ricerca - Università di Brescia

LOW LEVEL PROCESSING OF AUDIO ANDVIDEO INFORMATION FOR EXTRACTING THESEMANTICS OF CONTENTN. Adami, A. Bugatti, R. Leonardi, P. MiglioratiDEA - University of Brescia , Via Branze, 38 - 25123, Brescia, ItalyTel: +39 030 3715434; Fax: +39 030 380014e-mail: leon@ing.unibs.itAbstract - The problem of semantic indexing of multimedia documents is actually ofgreat interest due to the wide diffusion of large audio-video databases. In the first part ofthis paper we briefly describe some techniques used to extract low-level features (e.g., shotchange detection, dominant color extraction, audio classification, ...). Then the TOCAIframework for content description of multimedia material is presented, together with anapplication which implement it. Finally we propose two algorithms suitable to extract thehigh level semantics of a multimedia document. The first is based on finite-state machinesand low-level motion indices, whereas the second uses Hidden Markov Models.INTRODUCTIONEffective navigation through multimedia documents is necessary to enable wide-spread use and access to richer and novel information sources. Design of efficientindexing techniques to retrieve relevant information is another important requirement.Allowing for possible automatic procedures to semantically index audio-visual mate-rial represents a very important challenge. Such methods should be designed to createindices of the audio-visual material, which characterize the temporal structure of amultimedia document from a semantic point of view.In the first part of this paper we address the problem of low-level features extrac-tion showing different types of extraction algorithms, and some examples of possibleuse of them to perform simple tasks. Then a very powerful tool for browsing and re-trieval of interesting events inside a programme is shown, and the Description Scheme(ToCAI) on which is based is also described. Finally we show some interesting resultsin joint audio-video analysis based on the low-level features previously extracted.EXTRACTION OF LOW-LEVEL FEATURESWe have adopted several tools in order to obtain automatic feature extraction fordescribing an audio-visual document. These methods can be divided in two classes:audio and video.For video analysis we developed the following methods. Individual shot separation is achieved by extraction of editing effects betweenconsecutive camera records. This can be obtained by making use of the statis-tical independence of the two shots that are present on both sides of the editingeffect; in the case of dissolves, fade-in, or fade-our, refer to the algorithm pre-sented in [1]. Shot grouping into scenes is obtained by identification of peculiar alternationof visual patterns between consecutive shots, so as to recognize characteristicssituations such as dialogues, actions and so on. The visual correlation betweennon consecutive shots is established thanks to a vector quantization approach,which compares the codebooks associated to the individual shot patterns [2]. Dominant color extraction is very useful to have content-based retrieval forcolor, either for the whole image or for an arbitrarily shaped region. It is cal-culated using the color histogram and a measure of confidence which is high incases where pixels of dominant color represent the majority of the pixels in theobject, and it is low in the other cases.Considering the audio information, we developed some simple but effective meth-ods in order to have a significant segmentation of the audio stream that can be usefulfor browsing or retrieval of interesting events. Audio segmentation in homogeneous segments of speech and music is obtainedusing two different approaches: the first approach, based mainly on Zero Cross-ing Rate (ZCR) and Bayesian Classification, is very simple from a computa-tional complexity point of view [3]. The second approach, based on NeuralNetworks (specifically a Multi Layer Perceptron, MLP), allows better perfor-mance at the expenses of an increased computational complexity [4]. The average audio loudness extraction related to each shots. The shots havingthe higher audio loudness turned out to be very meaningful from the semanticpoint of view (but only in some kinds of programme: for instance, in a soccermatch there are goals, roar crowd after goals, and so on).TOCAI: A TOOL FOR BROWSING AND RETRIEVALAll the previous information can be used inside our application developed to pro-vide a tool for improved browsing and retrieval of multimedia documents. This tool isbased on a framework called ToCAI [5], [6], which is composed by two parts. The firstone, called ToC (Table of Contents), characterizes the temporal structure of a multi-media document from a semantic point of view at multiple levels of abstraction, so asto have a series of consecutive segments which are coherent in terms of the semanticof information at that level. The second one, called AI (Analytical Index), allows aneasy way to effectively retrieve relevant information, such as objects appearing in thevideo, or identify specific events of interest (e.g., a murder in a thriller movie or agoal in football match). To these ends, it is important that the objects or events arearranged in an appropriately designed index, according to criteria meaningful for theapplication context. A more detailed explanation can be found on the web [7].Figure 1: An interface view that shown the ToC and AI part.The ToC part has been built using the shot and scene separation algorithms ex-plained above. For the AI part some heuristic methods have been used in order toobtain objects ordering useful for retrieval operations. Another feature used is a mea-sure of the camera motion, which has been very useful in sport documents to extractthe shots with a player having fast movement.To improve the capabilities of this tool some approaches based on joint audio-video analysis have been studied, as showed in the next paragraph.JOINT AUDIO-VISUAL ANALYSISWe studied two promising approaches to extract more useful information fromaudio and video low-level features. The first approach uses MPEG motion vectors andfinite state machines in order to extract relevant events in a soccer game [8], whereasthe second one uses Hidden Markov Model to achieve a better scene segmentation [9].Soccer video indexing using MPEG motion vectorsThe algorithm is focused on analysing descriptors (mainly MPEG2 motion vec-tors) and related data to search for particular features in correspondence of interestingscenes in soccer games, for example the event of a scored goal. When a goal is scored,the following features are present: a fast pan; a fast zoom, used to focus the attention on a player or on the ball; a sequence of static frames, while the goal keeper is framed (this is optional); a scene cut.Level 0 Level 1 Level 2 GoalTime out 2Time out 1Fast panor zoomFast panor zoomNo motion Scene cutScene cutFigure 2: The final algorithm.The detection of the above mentioned features is implemented as follows: Camera motion parameters, represented by horizontal ”pan” and ”zoom” fac-tors, have been evaluated using a least-mean square method applied to P-framemotion fields. We have detected fast horizontal pan (or fast zoom) by threshold-ing the pan value (or the zoom factor). When a fast pan or a fast zoom is presentin 3 consecutive P-frames, level1 is set; Static frames are detected using average magnitude of motion vectors m: whenm takes low values for 3 consecutive P-frames, and level1 is present, level2 isset; Scene cuts are detected using a weighted sum of intra-coded Macro-Blocksand difference of average magnitude between the current and the future frame.When this sum takes a high value, a shot cut is present and if level1 or level2are present, level3 is set.The final algorithm is shown in Fig. 2.All levels are mutually exclusive; when a level is present and the event necessaryto set the next level doesn’t occur for a certain number of frames, the system is resetted(level0 is set). The algorithm to find penalties and corners works in a similar manner.The first simulation results show effectiveness of the proposed algorithm. The goalsare detected in the 90% of the considered cases. This percentage is quite good and inorder to improve the effectiveness of this algorithm also audio loudness informationis used. In fact, in this particular case, a sudden increase of audio loudness (due bothof the speaker and the crowd) can be a cue that a goal event is happening.Scene classification by Hidden Markov ModelIn this algorithm the focus is placed on providing tools for analyzing both audioand visual streams, for translating the signal samples into sequences of indices. Thefinal objective is to add a recognition task in order to identify statistical patterns of suchindices so as to enable a content-based audio-visual description (more at a semanticlevel) of the whole material (see figure 3).Audio-videoAudioclassificationShotsidentificationScenes revelation Scenes analysisFigure 3: The algorithm using HHM.First of all, the input stream is demultiplexed into the two components, audio andvideo. An independent segmentation and classification of the two channels, audioand video, represents the next step of the analysis. On one hand the audio streamis segmented into clips, and a feature vector is extracted from the low-level acousticproperties of each clip. On the other hand in the video analysis channel a featurevector is calculated by comparison of each couple of adjacent frames, in terms ofluminance histograms, motion vectors and pixel-to-pixel differences. Each sequenceof feature vectors extracted from the two streams is then classified by means of anHidden Markov Model (HMM), used in an innovative approach: the input signal isconsidered as a non-stationary stochastic process, modeled by an HMM in which eachstate stands for a different class of the signal. This defines an adaptive classificationscheme for which a set of new training algorithms was developed. Given a sequenceof unsupervised feature vectors, the correspondent most likely sequence of indicesidentifying particular signal classes could be generated using the Viterbi algorithm.A first approach to scene identification consists in the definition of four differ-ent types of scenes: dialogues, in which the audio signal is mostly speech and thechange of the associated visual information occurs in an alternated fashion (for exam-ple: ABAB...); stories, in which the audio signal is mostly speech while the associatedvisual information exhibits the repetition of a given visual content, to create a shotpattern of the type ABCADEFAG...; actions, when the audio signal belongs mostlyto one class (which is not speech), and the visual information exhibits a progressivepattern of shots with contrasting visual contents of the type ABCDEF...; finally con-secutive shots which do not belong to any one of the aforementioned scenes, but theirassociated audio is of a consistent type, are classified as generic scene. Once we havedefined these kinds of scenes, we can look for them in the time-aligned sequence ofdescriptors obtained as mentioned.A second and more general approach to scene classification is represented by astatistical pattern recognition analysis applying a clustering procedure on the basis ofthe sequences of descriptors obtained with a long-term analysis on multimedia data.This recognition system is very flexible and does not require defining the types ofscenes a priori.CONCLUSIONSIn this work we have presented some techniques for indexing of multimedia doc-uments, starting from low-level features and using joint audio-video information inorder to achieve an high level semantic information, useful for browsing and retrievalin audio-video databases. We have also described an application that uses the resultsobtained from the previous algorithms in order to allow an easy navigation through amultimedia document and some basic retrieval capabilities.References[1] N. Adami, R. Leonardi, ”Identification of editing effects in image sequences bystatistical modeling”, Proc. Picture Coding Symposium ’99, pp. 157-160, Portland,OR, U.S.A., Apr. 1999.[2] C. Saraceno, R. Leonardi, ”Identification of story units in audio-visual sequencesby joint audio and video processing”, Proc. International Conference on ImageProcessing 1998, Chicago, IL, U.S.A., Oct. 1998[3] A. Bugatti, R. Leonardi, L.A. Rossi, ”A video indexing approach based on audioclassification”, In Proc. International Workshop on Very Low Bitrate Video Coding’99, pp. 75-78, Kyoto, Japan, Oct. 1999.[4] A. Bugatti, A. Flammini, R. Leonardi, D. Marioli, P. Migliorati, C. Pasin, ”AudioClassification in Speech and Music: A Comparison of Different Approaches”, Proc.Workshop on Image Analysis For Multimedia Interactive Services 2001, pp. 153-158, Tampere, Finland, May 2001.[5] N. Adami, A. Bugatti, R. Leonardi, P. Migliorati, L.A. Rossi, ”The ToCAI De-scription Scheme for Indexing and Retrieval of Multimedia Documents”, Multime-dia Tools and Application Journal, Kluwer Academic Publishers, Vol. 14, No. 2,pp. 153-173, 2001.[6] N. Adami, A. Bugatti, R. Leonardi, P. Migliorati, L. Rossi, ”Multimedia docu-ments description by ordered hierarchies: the ToCAI description scheme”, Proc.International Conference on Multimedia and Expo 2000, New York, U.S.A., Aug.2000[7] http://www.extra.research.philips.com/euprojects/avir/.[8] A. Bonzanini, R. Leonardi, P. Migliorati, ”Semantic Video Indexing Using MPEGMotion Vectors”, Proc. EUSIPCO’2000, pp. 147-150, Tampere, Finland, Sept.2000.[9] F. Oppini, R. Leonardi, ”Audiovisual Pattern Recognition Using HMM forContent-Based Multimedia Indexing”, Proc. of Packet Video 2000, Cagliari, Italy,2000.

Low Level Processing of Audio and Video Information for Extracting the Semantics of Content

Adami, Nicola

Bugatti, Alessandro

Leonardi, Riccardo

Migliorati, Pierangelo

Low Level Processing of Audio and Video Information for Extracting the Semantics of Content

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università di Brescia

Archivio istituzionale della ricerca - Università di Brescia