This paper describes the design of an indexing system for a
video database. The system uses region-based manual an-
notations of keyframes to create models to automatically
annotate new keyframes also at the region level. The pre-
sented architecture includes user interfaces for training and
querying the system, internal databases to manage ingested
content and modelled semantic classes, as well as communi-
cation interfaces to allow the system interconnection. The
scheme is designed to work as a plug-in to an external Mul-
timedia Asset Management (MAM) system.Postprint (published version

Giró Nieto, Xavier

Marqués Acosta, Fernando

English

This paper describes the design of an indexing system for a

video database. The system uses region-based manual an-

notations of keyframes to create models to automatically

annotate new keyframes also at the region level. The pre-

sented architecture includes user interfaces for training and

querying the system, internal databases to manage ingested

content and modelled semantic classes, as well as communi-

cation interfaces to allow the system interconnection. The

scheme is designed to work as a plug-in to an external Mul-

timedia Asset Management (MAM) system

UPCommons

System architecture for indexing regions in keyframes

UPCommons. Portal del coneixement obert de la UPC

System Architecture for Indexing Regions in KeyframesXavier GiróTechnical University of Catalonia (UPC)Jordi Girona 1-3Barcelona, Catalonia (Spain)xavier.giro@upc.eduFerran MarquésTechnical University of Catalonia (UPC)Jordi Girona 1-3Barcelona, Catalonia (Spain)ferran@gps.tsc.upc.eduABSTRACTThis paper describes the design of an indexing system for avideo database. The system uses region-based manual an-notations of keyframes to create models to automaticallyannotate new keyframes also at the region level. The pre-sented architecture includes user interfaces for training andquerying the system, internal databases to manage ingestedcontent and modelled semantic classes, as well as communi-cation interfaces to allow the system interconnection. Thescheme is designed to work as a plug-in to an external Mul-timedia Asset Management (MAM) system.Categories and Subject DescriptorsH.2.4 [Database Management]: SystemsGeneral TermsDesignKeywordsindexing, architecture, MAM, keyframe, region1. INTRODUCTIONContent producers and TV broadcasters have migrated dur-ing the last decade from analogue to digital systems, turningtheir shelves full of tapes into large multimedia databasescalled Media Assets Management (MAM) systems. Thesesystems have enormously improved their capacity of ingest-ing, producing and delivering new content but, at the sametime, have increased their need of generating metadata as-sociated to it. This situation, combined with the high costof manual annotation, has raised the interest in automaticcontent indexing.This paper presents the architecture of an indexing systemaimed at helping documentalists in their task of indexing thegrowing amount of audiovisual content managed by MAMs.The system is designed as a plug-in for a MAM and acts asan external annotator that generates keywords for the vi-sual content. The analysis is performed on keyframes previ-ously extracted from each video asset and is based on imageprocessing techniques that process keyframes at the regionlevel. The design includes the elements necessary to trainthe system in generating new keywords automatically.The document is divided in two parts. Firstly, the descrip-tion of the six basic elements in the architecture and, sec-ondly, the description of three basic user operations that canbe performed with these elements. In all cases, the extrac-tion of keyframes from video sequences is considered to beperformed at a previous stage.2. ARCHITECTUREThe system is composed of six different elements which mayrun in different and remote machines.2.1 Content DatabaseAll keyframes from video assets stored in the MAM are an-alyzed by the indexing system to create their region-basedrepresentation. This pre-processing corresponds to the ex-traction of visual descriptors from the data, a task that typ-ically implies an important computational effort. As thesefeatures will be repeatedly assessed by different queries inthe future, they are generated at ingest time and stored forfuture use in the Content DB2.2 Knowledge DatabaseThe keywords used by the MAM to index its contents areinternally treated by the system as descriptions of anotherentity called semantic class, which can represent either anobject, person, location or event. All considered classes arestored in a Knowledge DB and organized in one or severalontologies that represent their relationships. In addition tothe mentioned keywords, this database also mantains an in-dex of annotated instances for each semantic class .2.3 Model DatabaseEach semantic class has one or more data models built af-ter a training process. These models contain the classifierparametes and are typically learnt from the manual annota-tions of a small portion of the content database.2.4 Instance DatabaseEach annotated instance is stored in the Instances DB, spec-ifying the video asset, keyframe, semantic class and degree1of confidence on the annotation. Annotations can be gener-ated manually or automatically and have an author associ-ated, whether a human (manual annotations) or a machine(automatic annotations).2.5 User InterfacesUsers interact with the system through graphical interfacesfrom which they can perform the following basic operations:annotation of key-frames at region-level, revision of auto-matic annotations generated by the system, management ofsemantic ontologies and search based on text or visual ex-amples.2.6 Communication InterfacesEach element in the system includes a communication in-terface that establishes and controls the data transmissions.Two different types of protocols are used depending on thedata nature: images (keyframes and partitions) are trans-mitted over FTP connections and the remaining data is ex-pressed in MPEG-7/XML [1] metadata sent over SOAP pro-tocol3. STUDY CASESThe defined elements aim at offering solutions to three studycases that cover the most important applications of the sys-tem.3.1 Content ingestWhen a MAM sends a new keyframe to be analyzed, this ispre-processed and added to the Content DB. The presentedimplementation segments the key-frame to generate an im-age partition and a multiscale segmentation by building aBinary Partition Tree (BPT) [2] on the top of it. After-wards, a set of visual descriptors are extracted from eachregion in the BPT.After the pre-processing, keyframes are analyzed for its au-tomatic annotation by considering all entries in the ModelDB. The detected instances, if any, are added to the In-stances DB and the results of the analysis are sent back tothe MAM under the form of the keywords associated to thedetected instances. Additionally, the returned metadata canbe enriched with a degree of confidence on the results or in-formation related to the visual appearance and localizationof the instances.3.2 Manual annotationUsers can manually annotate regions from the keyframesthrough a graphical user interface (GUI), a task that gener-ates high-confidence metadata while providing positive andnegative labels for training classifiers. The annotation pro-cess begins by selecting a semantic class from the KnowledgeDB or defining a new one from the same GUI. Secondly, akeyframe from the Content DB is chosen by the user andretrieved together with its partition and BPT. Finally, usermarks the regions that represent an instance of the seman-tic class using the BPT to navigate through the keyframeregions, as described in [3].After each manual annotation, the instances are added tothe Instances DB and used to create or update a modelof the annotated class. Model learning algorithms use vi-sual descriptors from the annotated instances to train itsassociated classifiers. These visual features may already beamong the descriptors computed during keyframe ingest andavailable at the Content DB. When the training process isfinished, the new model must be evaluated on the wholeContent DB. Due to the high computational requirementsof this task, this process is only applied immediately on thepreviously positive annotations of the class, while the therest of the database normally is explored later during peri-ods of low usage of the system.3.3 Query by regionIn addition to the common query by keyword supported bymost of MAMs, the presented system offers the option ofquerying the database with visual content and obtain a rank-ing of keyframes representing regions similar to the query. Inthis case, the user formulates its query by selecting a regionwith a GUI similar to the one used for manual annotation.The descriptors associated to the query region are comparedwith those in the Content DB and a similarity measure isgenerated for each pair to finally build an ordered list. If thequery keyframe is not in the Content DB, it is previously in-gested to generate its partition, BPT and extract its genericdescriptors.4. CONCLUSIONSThis document has presented a system architecture that sat-isfies the basic requirements of an automatic indexing systemfor video content. The proposed system works at a tempo-ral resolution of a keyframe and at a spatial resolution ofa region. The design tries to maximize the modularity inorder to facilitate its final implementation in one or multi-ple machines as well as allowing an easy integration withan already running MAM. The current design presents twobottlenecks in terms of computation time. Firstly, duringthe evaluation of all Model DB entries on each new ingestedkeyframe; and secondly, in the re-exploration of the wholeContent DB every time a model is updated. For this rea-son, future work will focus on techniques for indexing andexploring regions organized in hierarchical structures suchas BPTs.5. ACKNOWLEDGMENTSThis work was partially founded by the Catalan Broadcast-ing Corporation (CCMA) and Mediapro through the Span-ish project CENIT-2007-1012 i3media, and by TEC2007-66858/TCM PROVEC project of the Spanish Government.6. REFERENCES[1] P. Salembier. B.S. Manjunath and E. T. Sikora, editors.Introduction to MPEG-7: Multimedia ContentDescription Interface. Wiley, 2002.[2] P. Salembier and L. Garrido. Binary partition tree asan efficient representation for image processing,segmentation and information retrieval. IEEE Trans.on Image Processing, 9(4):561–576, April, 2000.[3] X. Giro´, N. Camps and F. Marque´s. Region-basedannotation tool using partition trees. In Poster andDemo Proceedings of the 2nd International Conferenceon Semantic and Digital Media Technologies,SAMT’07, Genova, Italy, December, 5-7 2007.2

https://upcommons.upc.edu/bitstream/2117/10893/2/samtPoster.pdf

System architecture for indexing regions in keyframes

Abstract

Similar works

Full text

Available Versions

UPCommons

UPCommons. Portal del coneixement obert de la UPC