International audienceThis paper describes the PACE experimental multimedia application that aims at providing automatic tools for web browsing of television program collections; experimentations are currently in progress with a fifty-four Le Grand Échiquier show collection. PACE has been built with the FERIA framework and relies on multiple automatic analysis tools. It is generic enough to easily adapt to other collections. Emphasis is made on the new audiovisual documents description language FDL as it is the core part of FERIA, with a particular attention paid on how it operates in PACE

Brunie, Vincent

Caillet, Marc

Carrive, Jean

Roisin, Cécile

INRIA a CCSD electronic archive server

HAL Id: inria-00423351https://hal.inria.fr/inria-00423351Submitted on 9 Oct 2009HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.PACE: an Experimental Web-Based AudiovisualApplication using FDLMarc Caillet, Jean Carrive, Vincent Brunie, Cécile RoisinTo cite this version:Marc Caillet, Jean Carrive, Vincent Brunie, Cécile Roisin. PACE: an Experimental Web-Based Au-diovisual Application using FDL. Web Document Analysis 2005, Aug 2005, Singapour, Singapore.￿inria-00423351￿PACE: an Experimental Web-Based Audiovisual Application using FDLMarc Caillet, Jean Carrive, Vincent BrunieInstitut National de l’Audiovisuel (INA)Audiovisual Content Description4 avenue de l’Europe,94366 Bry-sur-Marne Cedex, France{mcaillet,jcarrive,vbrunie}@ina.frCécile RoisinINRIA Rhône-Alpes & UPMFWAM Project655 avenue de l’Europe, Montbonnot,38334 St-Ismier Cedex, France{marc.caillet,cecile.roisin}@inrialpes.frAbstractThis paper describes the PACE experimental multime-dia application that aims at providing automatic tools forweb browsing of television program collections; experimen-tations are currently in progress with a fifty-four ”Le GrandÉchiquier” show collection. PACE has been built with theFERIA framework and relies on multiple automatic analysistools. It is generic enough to easily adapt to other collec-tions. Emphasis is made on the new audiovisual documentsdescription language FDL as it is the core part of FERIA,with a particular attention paid on how it operates in PACE.1. IntroductionDigitization is now coming as a required step in the pro-duction or archiving process of audiovisual contents. More-over, there is a growing number of television programs thatare numerically produced. As a result, audiovisual docu-ments are no more analog tapes, they are computer files.Their contents can be thus accessed without knowing nei-ther their physical location nor their physical or logicalstructure. Queries as well as access to audiovisual contentonly require descriptions of the content. That is where vir-tuality lies. Moreover, it is possible to access audiovisualcontents from different points of view depending of whatis sought for. With such new access modes, new multime-dia applications of a great interest for audiovisual resourcesholders are rising.This paper focuses on the PACE1 experimental applica-tion that aims at automatically publishing television pro-gram collections on the web. Section 2 describes its ob-jectives and functionalities as well as the FERIA2 projectwhich PACE has been developed with. For virtual audiovi-sual contents to be accessed through descriptions, a descrip-tion language is needed. Section 3 briefly presents a new au-1Automatic Publishing of a Television Program Collection.2Framework for experimentation and industrial production of multime-dia applications.diovisual document description language, FDL3, that formsthe core part of the FERIA framework. Section 4 describeshow it operates in PACE and what PACE is gaining fromit. Conclusion and future developments are the subject ofSection 5.2. The PACE applicationINA4 has been archiving and indexing broadcastedFrench television and radio programs for thirty years andhas thus to manage huge audiovisual resources. Accord-ing to the requests it is daily receiving from individuals, themost promising market for television and radio programsarchives lies in excerpts. For this market to be profitable,low cost and generic audiovisual contents browsing appli-cations must be provided. That is exactly what the experi-mental PACE application is aiming at.PACE provides a generic way to automatically publishtelevision program collections on the web. It also allowsindividuals to navigate by queries through these collec-tions and thus to easily discover excerpts. Experimenta-tions are currently in progress with a fifty-four ”Le GrandÉchiquier”5 show collection. Anyone who is seeking foran excerpt of one of these shows through PACE is providedwith many browsing methods: he is first able to consider ei-ther the whole collection or a specific show; he is then ableto ask for the faces of the guests that appear in the collec-tion or in the specific show he previously selected, or askfor the faces of guests that may have said words of his in-terest, or even for broached topics; he is finally able to askfor sequences in which a specific guest appears either as aperformer or as an interviewee, or sequences about a spe-cific topic of his interest. And the process may go on andon as our excerpts seeker is now able, at this stage of hisresearch, to ask for the faces of guests that appear in — orthe broached topics of — the returned sequences and so on.3Feria Description Language.4French National Institute of Audiovisual.5One of the most famous French variety show (from 01/72 to 12/89).It is important to note that the queries are hard-wired sothat the excerpts seeker does not need toa priori know whathe is seeking for. Thisparti pris allows him to discover theaudiovisual contents of the collection by strolling throughit, without being required to know any query language.PACE is being developed within the FERIA frameworkwhose purpose is to design low cost multimedia applica-tions based on automatic analysis (some industrial applica-tions, such as MANREO or KINOMAI6, are partly sharingsome features with FERIA). Figure 1 depicts its global ar-chitecture.Figure 1. PACE architecture.INA archivists annotate the documents with cataloguinginformation,e.g. the director’s name. The analysis engineruns the automatic analysis tools — video and audio analy-sis (e.g.[2]), and audio classification (e.g.[5]) — accordingto an analysis graph (see figure 2). It produces many differ-ent viewpoints on a given television program, and thus asmany descriptions. Both human- and computer-generateddescriptions are processed by the publishing engine to au-tomatically produce a web site. This architecture makesPACE generic enough to easily adapt to other television pro-gram collections.3. FDL, a new audiovisual documents descrip-tion languageFDL forms the core part of the FERIA framework asits audiovisual document description language. How camewe proposed a new description language whereas describ-ing audiovisual documents is what theMultimedia ContentDescription InterfaceISO standard, known as MPEG-7 ([3]and [4]), is designed for? While this question is extensivelydiscussed in a previous work of our own ([1]), the followingdraws a general survey of it.3.1 FDL, an alternative to MPEG-7Requirements for an audiovisual description languageare resulting from the specific needs of multimedia appli-cations such as PACE:• expressiveness power: taxonomic hierarchy of de-scriptors as a way to express their semantics; abilityto create a new descriptor by structuring other descrip-tors; link to the media independently of its physicallocation;6see respectively www.netia.net/index.php and www.kinomai.com• descriptor processing: specification of descriptors asdescription classes that can instantiated; controlled ex-tensibility and modularity;• platform- and application-independent syntax.Because our own requirements are close to those ofMPEG-7, we first evaluated this standard as an applicantto fulfill them and came to the conclusion that it doesn’tmeet them all. One of the major drawback of MPEG-7 liesin its inability to taxonomically hierarchize the descriptorsand thus its inability to express their semantics. This draw-back is due to the choice of XML Schema as MPEG-7 DDL(Definition Description Language). Other drawbacks areMPEG-7 non-modularity and non-extensibility — or ratherMPEG-7 inability to validate extensions of its descriptionschemes. These drawbacks harshly shake MPEG-7 deep-est foundations; the problem is too important to be solvedby simply modifying the standard. We thus opted for aparadigm switch: we switched from a documentary one inwhich the description language allows the syntactic expres-sion of audiovisual document annotations (e.g. MPEG-7)to a knowledge representation one in which audiovisual de-scriptors are objects organized in a taxonomic hierarchy thatholds semantics.3.2 FDL, a short descriptionFDL is an object language. It can be thought about asa meta-language that allows the definition of application-specific audiovisual document description languages. Allof these FDL-languages share a few constants:• Only three semantically related generic descriptors onthe basis of which other descriptors may be built:Descriptor is the root descriptor which any otherdescriptors inherits from;TemporalDescriptor in-herits properties fromDescriptor and adds a tem-poral location property;SpatialDescriptor inher-its properties fromTemporalDescriptor and adds aspatial location property.• Extensibility mechanisms: inheritance of propertiesand composition of descriptors. These mechanismsare fully under FDL control so that any user-defineddescriptor can be validated.Such a short number of descriptors truly contrasts withMPEG-7 six hundred multimedia description schemes. To-gether with the extensibility mechanisms, they provide FDLwith a major asset: they allow to freely design consistent de-scriptors that fit any application specific needs, thus makingFDL readily adaptable to new multimedia applications tocome. Using MPEG-7, instead, would have led to either tryto adapt to predefined descriptors or to design extensions ofthem that could not be validated.Other FDL constants are: link to the media, spatial andtemporal localisation types, basic types, matrix of any di-mension type. Altogether, they allow to define: descrip-tors as description classes (DCs) and descriptions (Ds) asinstances of description classes.XML is the syntax for FDL. It has been opted for in or-der to facilitate the storage and the management of descrip-tion classes and descriptions, and to allow the use of well-established standard XML tools to perform some validationtasks.4. FDL for PACEFDL is used in FERIA to syntactically and semanticallyexpress the descriptors. Figure 2 depicts the analysis graphthat leads to PACE. The directed edges outline the FDL de-scriptions flow: some FDL descriptions are used as inputsto analysis tools, some others are directly used by PACE.Figure 2. PACE analysis graph (KF stands for Key Frame).This section focuses on the results of the shot segmen-tation tool. It describes how theShotSegmentation-ToolResult descriptor and one of its instance are codedin FDL, and how they are validated. It ends with a discus-sion about what PACE is gaining from FDL.4.1 The ShotSegmentationToolResult descriptionclass and an instanceFigure 3 shows the UML modelling schema of theShotSegmentationToolResult description class.Figure 3. UML schema of ShotSegmentationToolResult.The AutomaticToolResult is an abstract descriptionclass,i.e. it cannot be instantiated, that subsumes the audioand video description classes that are used for PACE. Thescript below shows a sample of its FDL code:<DescriptionClass ... ><DescriptionClassHeader><Urn>urn:x-feria:dc:automatictoolresult</Urn></DescriptionClassHeader><DescriptionClassBody><Descriptor id="x-0"><Name>AutomaticToolResult</Name><Parent>fdl:TemporalDescriptor</Parent><DescriptorAttributes>Abstract</DescriptorAttributes></Descriptor></DescriptionClassBody></DescriptionClass>An FDL description class is divided into two distinctparts: the header specifies the urn of the descriptor; the bodydefines the descriptor itself which carries anid attribute (itsid is x-0 in the example above) so that it can be referred toby other descriptors.Inheriting. The Parent element creates an inheritancerelationship between theAutomaticToolResult descrip-tor and thefdl:TemporalDescriptor FDL generic de-scriptor. Note that every user-defined descriptor must in-herit from one of the three FDL generic ones to be deemedto be a new FDL descriptor.The VideoToolResult description class subsumes allvideo description classes PACE makes use of. The first FDLsample code below shows how theVideoToolResult de-scriptor inherits from theAutomaticToolResult descrip-tor:<DescriptionClass ...xmlns:atr="urn:x-feria:dc:automatictoolresult" ... >... <Parent>atr:x-0</Parent> ...As for the previous descriptor, the inheritance link is cre-ated by theParent element. In the sample code above, itis valued toatr:x-0 : on the left side,atr is a shortcutthat refers to theAutomaticToolResult description classthrough its urn; on the right side,x-0 refers to the id of thedescriptor whichVideoToolResult inherits from.Adding a new property. The second FDL sample codeof theVideoToolResult description class shows how theFrameWidth property is added:<Property id="p-0"><Name>FrameWidth</Name><Range type="integer"/></Property>Structuring. The following FDL sample code of theShotSegmentationToolResult description class em-phasizes how structures are being used:<DescriptionClass ...xmlns:sgmt="urn:x-feria:dc:segment" ... >...<Structure id="s-0" use="required"><Name>Segments</Name><Type>fdl:TemporalStructure</Type><Descriptor id="d-1" import="sgmt:d-0"/></Structure>...The Type element points out that theSegments struc-ture is of typefdl:TemporalStructure , i.e. that it isa list of temporal descriptors without any other constraint.The Descriptor element that follows forces upon thestructure it is enclosed within to only accept temporal de-scriptors of a specific type. Theimport attribute valued tosgmt:d-0 specifies that these descriptors must beSegmentones, and that theSegment description class (that subsumesShot , Transition and Cut as depicted in figure 3; itsFDL code is not given in this paper) is located elsewhere.Instancing. Finally, the sample code below is an instanceof theShotSegmentationToolResult descriptor whereall the properties and structure are instantiated :<fdl:Description ...>...<fdl:DescriptionBody><ShotSegmentationToolResult id="shotsegment-0001"><fdl:DescribedEntityUrn>urn:x-feria:doc:ina:CPB81050169</fdl:DescribedEntityUrn><fdl:Begin>00:00:00:00000000</fdl:Begin><fdl:End>00:00:30:00000000</fdl:End><FrameWidth>352</FrameWidth><FrameHeight>288</FrameHeight><Segments id="segments"><s:Shot id="shot-1"><fdl:Begin>00:00:00:00000000</fdl:Begin><fdl:End>00:00:02:00000000</fdl:End></s:Shot><t:Transition id="trans-1"><fdl:Begin>00:00:02:00000000</fdl:Begin><fdl:End>00:00:03:00000000</fdl:End></t:Transition>...4.2 ValidationFDL has been implemented within the .NET framework.Briefly, a description class parser dynamically compiles de-scription classes to dynamic libraries, and a descriptionparser loads the relevant classes and dynamically instan-tiates them. Both description classes and descriptions arevalidated while being parsed, as figure 4 illustrates it for theShotSegmentationToolResult (SSTR).Figure 4. The validation process.XML schema is used to syntactically constraint the de-scription classes, and to validate a part of the logical struc-ture and data types of the descriptions. FDL completes thevalidation process by taking charge of what XML schema isunable to express. For example: checking that the descrip-tor is a direct or undirect descendant of one of the three FDLgeneric descriptors, checking that the structures constraintsare fulfilled. TheSSTR.xsd schema is automatically gen-erated while theSSTRdescription class is parsed.4.3 What is PACE gaining from FDL?At a given step of his investigation, the excerpts seekerwe considered in section 1 is provided with the ability toask either for sequences of any type, or for performance orinterview ones. Because the FDL underlying object modelallows to model both performance and interview sequencesas sub-classes of the sequence class (so that instances ofthese two classes are also instances of the sequence class;this semantic link cannot be expressed using MPEG-7), itis easy for PACE to answer to the corresponding queries.FDL is also able to validate extensions of its predefineddescriptors, whereas MPEG-7 is not. These are two greatfeatures because they allow us to design our own consis-tent descriptors. These descriptors may have been designedusing MPEG-7 but without semantical links between themand without the ability to be validated.Moreover, FDL is fully modular. So that bothtools and FDL descriptors are reuseable in FERIA todesign other multimedia applications. For example,FaceDetection , StudioKF , SpeechDetection andSpeaker Segmentation and Recognition are alsopart of the automatic analysis tools sequence that leads tothe FIDELIO/ALTO7 experimental applications.5. Conclusion and future worksThis paper showed how PACE has been designed withthe FERIA framework and what it is gaining from FDL. Ev-ery analysis tool works correctly and produces descriptionsin FDL. The analysis graph has been ”manually” validated.However, the PACE application is still work in progress. In-tegration of the tools within the FERIA framework is left tobe done. Moreover, FDL itself is evolving to enhance thedefinition and control of structures. For example, in PACE,more constrained structures would be of a great usefulnessfor the Segments structure to constrain its descriptors notto time-overlap. Dealing with such constraints is going tobe an important part of our short term future work.References[1] M. Caillet, J. Carrive, and C. Roisin. Description des docu-ments audiovisuels : s’affranchir des limitations de mpeg-7.In Proceedings of MetSI05, pages 69–78, May 2005.[2] B. Fauvet, P. Bouthemy, and P. Gros. A geometrical key-frame selection method exploiting dominant motion estima-tion in video. InProceeding of CIVR, pages 419–427, 2004.[3] J. M. Mart́ınez, R. Koenen, and F. Pereira. Mpeg-7: Thegeneric multimedia content description standard, part 1.IEEEMultimedia, 9(2):78–87, April–June 2002.[4] J. M. Mart́ınez, R. Koenen, and F. Pereira. Mpeg-7: Thegeneric multimedia content description standard, part 2.IEEEMultimedia, 9(3):83–93, July–September 2002.[5] J. Pinquier and R. André-Obrecht. Audio classification toolsfor multimedia indexing. InProceedings of CBMI, 2005.7FIDELIO enhances an opera show with metadata, ALTO provides away to use thse metadata in an interactive television program broadcast.

PACE: an Experimental Web-Based Audiovisual Application using FDL

https://hal.inria.fr/inria-00423351/document

PACE: an Experimental Web-Based Audiovisual Application using FDL

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server