Petkovic, M.

English

NARCIS 

Content-based video retrieval

University of Twente Research Information

Content-based video retrieval03HWNRYL`Centre for Telematics and Information Technology, University of TwenteP.O. Box 217, 7500 AE Enschede, The NetherlandsPhone: +31 53 4893725, Fax: +31 53 4894524E-mail: milan@cs.utwente.nl1. IntroductionRecent advances in multimedia technologies allow thecapture and storage of video data with relativelyinexpensive computers. Furthermore, the newpossibilities offered by the information highways havemade a large amount of video data publicly available.However, without appropriate search techniques allthese data are hardly usable. Users are not satisfiedwith the video retrieval systems that provide analogueVCR functionality. They want to query the contentinstead of the raw video data. For example, a useranalysing a soccer video will ask for specific eventssuch as goals. Content-based search and retrieval ofvideo data becomes a challenging and importantproblem. Therefore, the need for tools that canmanipulate the video content in the same way astraditional databases manage numeric and textual datais significant.This paper presents our approach for content-basedvideo retrieval. It is organised as follows. In the nextsection, we give an overview of related work. Thethird section describes our approach with emphasis onthe video modelling as one of the most criticalprocesses in video retrieval. The architecture of acontent-based video retrieval system is presented inthe fourth section. The fifth section draws conclusion.2. The state of the art overviewAs we already mentioned, modelling the videocontent is one of the most important tasks in videoretrieval. In the literature, video content is approachedat different levels: raw data, low-level visual contentand semantic content. The raw video data consists ofelementary video units together with some generalvideo attributes such as format, frame rate etc. Low-level visual content is characterised by visual featuressuch as colour, shapes, textures etc. Semantic contentcontains high-level concepts such as objects andevents. The semantic content can be presentedthrough many different visual presentations usingdifferent sets of raw data. It is obvious thatrequirements for the extraction of these contents aredifferent. The process of extracting the semanticcontent is the most complex, because it requiresdomain knowledge or user interaction, whileextraction of visual features can be often doneautomatically and it is usually domain independent.Extensive research efforts have been made withregard to the retrieval of video and image data basedon their visual content such as colour distribution,texture and shape [1]. These approaches are mainlybased on similarity measurement. Examples includeVisualSEEk [2], Photobook [3], Blobworld [4], aswell as Virage video engine [5], CueVideo [6] andVideoQ [7] in the field of video. The image retrievalsystems allow a user to make queries based on visualimage content – properties such as colour percentages,colour layout and textures occurring in the imagesusually by using instances of prior matches (query byexample). Some of these systems use the spatialinformation and allow the user to make queries bysketching the layout of colour regions, or byproviding the URL of a seed image. First approachesin video retrieval just added the functionality forsegmentation and key frame extraction to existingimage retrieval systems. After key-frame extraction,they just apply similarity measurement on them basedon low-level features. This is not satisfactory becausevideo is temporal media, so sequencing of individualframes creates new semantics which may not bepresent in any of the individual shots. Furthermore,choosing the key-frames is still a challengingproblem.Query by example approaches are suitable if a userhas a similar image at hand, but they would notperform well if the image is taken from a differentangle or has a different scale. The naive user isinterested in querying at the semantic level rather thenhaving to use features to describe his concepts.Nevertheless, good match in terms of the featuremetrics may yield poor results (multiple domainrecall, e.g. a query for 60% of green and 40% of bluemay return an image of a grass and sky, a green boardon a blue wall or a blue car parked in front of a park,as well as many others).Modelling the semantic content is far more difficultthen modelling the low-level visual content of avideo. At the physical level video is a temporalsequence of pixel regions without direct relation to itssemantic content. Therefore, it is very difficult toexplore semantic content from the raw video data. Inaddition to that, if we consider multiple semanticmeaning such as metaphorical, associative, hidden orsuppressed meaning, which the same video contentmay have, we make the problem even more complex.The simplest way to model the video content is byusing free text manual annotation. An example is‘stratification’ approach [8] with a few extensions [9,10]. Some other approaches [11, 12] introduceadditional video entities, such as objects and events,as well as their relations, that should be annotated,because they are subjects of interests in video.Another way to model the video entities assumesusing spatio-temporal relations. The concept of videoobject can be associated to the sub-frame region thatconveys useful information, while spatio-temporalrelations among these objects can be defined asevents. Modelling of these high-level concepts(objects and events) gives the possibility to describeobjects in space and time and capture movements ofobjects. As humans think in term of events andremember different events and objects after watchingvideo, these high-level concepts are the mostimportant cues in content-based retrieval. Let’s takeas an example a soccer game, humans usuallyremember goals, interesting actions, red cards etc. Afew attempts to include these high-level concepts intovideo model are made in [13, 14, 15].The distinction, we made regarding modelling thevideo content, makes clear two important things. Onthe one hand, feature-based models use automaticallyextracted features to represent the content of a video,but they do not provide semantics that describes high-level concepts of video, such as objects and events.On the other hand semantic models usually use freetext/attribute/keywords annotation to represent thehigh-level concepts of the video content that results inmany lacks. The annotation is tedious, subjective andtime consuming. One of the major limitations of thisapproach is that search process is based mainly on thepredefined attribute information, which are associatedby video segments manually by human or(semi)automatically in the process of annotation.Obviously, an integrated approach, that will provideautomatic mapping from features to high-levelconcepts, is the challenging solution.3. The third way: Concept inferencingIn order to overcome the problem of mapping featuresto high level concepts we propose a layered video datamodel that has the following structure. The raw videodata layer is at the bottom. This layer consists of asequence of frames, as well as some video attributes,such as compression format, frame rate, number ofbits per pixel, colour model, duration, etc. The nextlayer is the feature layer that consists of domain-independent features that can be automaticallyextracted from the raw data. There are two types ofvideo features: static features characterising a stillimage (frame), such as shapes, textures, colourhistogram, etc., and dynamic features characterisingframe sequences, such as temporality, motion, etc.The features are assigned to regions. We define aregion as a contiguous set of pixels that ishomogeneous in one or more features. As we can seein [16] a region could be automatically extracted andtracked. The concept layer is on the top. It consists oflogical concepts that are subject of interest to users orapplications. Automatic mapping from the raw videodata layer to the feature layer is already achieved, butautomatic mapping from the feature to the conceptlayer is still a challenging problem. We simplify thisproblem by dividing the concept layer into the objectand event layer.The object layer consists of entities (logical concepts)that can be assigned to one or more regions. Wedefine a video object as a collection of video regions,which have been grouped together under some criteriadefined by the domain knowledge. These objectsshould also satisfy some conditions such that theyshould be semantically consistent, representing onereal-world object, and subject of interest to users orapplications. Some examples of video objects are aspecific player or the ball in a soccer game or aspecific car in a car-race video. As we can see in theliterature [17, 18, 19] automatic detection of videoobjects (sub-frame entities) in a known domain isfeasible. For this purpose, we proposed the objectgrammar that consists of rules for object extractions.A simplified example of an object rule in the soccerdomain could be “if the shape of a region is round, thecolour is white and it is moving, that object is a ball”.The event layer is the highest layer in our model. Itconsists of events, which describe object interactionsin the spatio-temporal manner. For the automaticmapping from the object layer to the event layer, wepropose the event grammar that consists of rules fordescribing event types. The event type could bedescribed using object types, audio segment types,spatio-temporal and real-world relations. For example,in the soccer domain, if the ball object type is insidethe goalpost object type for a while and this isfollowed by very loud shouting and a long whistle,that might indicate that someone has scored a goal,which should be recognised as a goal event.The main advantage of the proposed layered videodata model is that it provides a framework forautomatic mapping from features to concepts. Thisapproach bridges the gap between domainindependent features, such as colour histograms,shapes, textures and domain dependent high-levelconcepts such as objects and events. The audiocomponent is integrated in the model to provideadditional information that can be critical to theperception and understanding of video content.The formal definition of the proposed video model, aswell as a few modelling examples can be found in[20].4. The architecture of content-based videoretrieval systemThe proposed architecture for a content-based videoretrieval system is shown in Fig. 1. The process ofdatabase population is shown with dashed lines, whilequerying is shown with solid lines. The raw videodata is stored in the file system, while the storageserver is used to store video content meta data andindexes. In the process of the database population, thefeatures, objects, and events that are specified bysystem administrator, are extracted. Indexes and meta-data are put in the storage server and videos in the filesystem. Most queries are resolved directly in thestorage server, but if the query comprises somethingthat has not been already extracted the extractors dothat dynamically.- Feature- Object- EventExtractorVideoLoaderQueryInterfaceQueryComposerStorage server- features- objects- eventsRawvideodataDB population QueryQueryresultsFile systemFig. 1. The system architectureThe implementation platform for the storage servershould be chosen very carefully. In addition tostorage, it should support efficient management andhomogeneous querying of features, objects andevents. For example, the storage server should becapable to deal with distance functions in featurespaces to perform similarity measurements. In orderto enable modelling and querying objects and eventsusing the grammars, it should support a basic set ofspatio-temporal relations like ones defined in [21, 22].As far as temporal relations are concerned, point andinterval data type should be supported to representframes and frame sequences respectively. Each objectand event has, as an attribute, a set of intervals, i.e.frame sequences, where it occurs. The basic relationsof interval and point temporal algebra, the mappingbetween them, as well as operations on the intervaldata type, such as intersect and union, have to bedefined.Having these requirements in mind, we chose theMoa/Monet platform for the prototype of the videoretrieval system that is currently under development.At the physical level, we use Monet [23] – anextensible parallel database kernel developed at theCWI and the University of Amsterdam. Theadvantages of Monet, that have influence on ourdecision to chose it for physical level, are its mainmemory query execution, extensibility andparallelism. At the logical level, we use the Moaobject data model and algebra [24] developed at theUniversity of Twente. The Moa data model is astructural object data model. It accepts all base typesof the underlying physical storage system and allowstheir orthogonal combination using the structureprimitives set, tuple and object. The chosen platformis extensible and we believe that our goal, the videodatabase system that allows retrieval based on thevideo content, can be reached easier.At the conceptual level, the video data model and aquery language that is especially intended to supportvideo retrieval are defined. This level provides easy touse object and event descriptions. A user can definehis concepts at this level. They will be automaticallytranslated to the Moa object algebra at the logicallevel and then again automatically to the Monet’salgebra at the physical level.The basic video model should be accompanied by adomain model. It means that fundamental classes,which deal with the video structure, as well as withmeta content (video object types, event types, etc.),are defined. When the domain for the video retrievalis chosen, classes that model the domain itself have tobe added. In this way, we keep a general approach andavoid the problem of modelling the whole world.5. ConclusionThe proposed approach integrates feature-based andannotation-based retrieval approaches taking their bestcharacteristics and avoiding their limitations anddrawbacks. The main advantage of the model is that itprovides a framework for automatic mapping fromfeatures to semantic concepts, integrating audio andvideo primitives. This approach bridges the gapbetween domain independent features, such as colourhistograms, shapes, textures and domain dependenthigh-level concepts such as objects and events. Thefour-layer structure of our video model guides theprocess of translating raw video data into efficientinternal representation that captures video semantics.Easier description of video content is supported by therobust object and event grammars that can be used forspecifying even very complex objects and events. Theformalisation of an event as a spatio-temporaldescription of object interactions results in easiercapturing of high-level concepts of video content, andallows queries that are closer to the user way ofthinking (users’ cognitive maps of a video).The proposed model has formal foundation andlayered structure that enables using of differenttechniques at different layers as well as combiningdifferent techniques at the same layer. This allowsconcurrent use of for example a MPEG-4 objectextractor, other object extractors based on objectrecognition, and the object grammar at the objectlayer. The model also supports flexible videosegmentation using high-level concepts. This allows auser to make different logical segmentation of thesame raw data dynamically, building differenthierarchies.References[1] P. Aigrain, H. Zhang, D. Petkovic, Content-based Representation and Retrieval of Visual Media:A State-of-the-Art Review, Multimedia Tools andApplications, Kluwer Academic Publishers, 3(3),1996, 179-202.[2] J. R. Smith, S-F. Chang, “VisualSEEk: A FullyAutomated Content-Based Image Query System”,ACM Multimedia Conference, Boston, MA,November 1996.[3] A. Pentland, R. W. Picard, S. Sclaroff,“Photobook: Content-Based Manipulation of ImageDatabases”, Int. J. Computer Vision, 18 (3), pp. 233-254.[4] C. Carson, M. Thomas, S. Belongie, J. M.Hellerstein, J. Malik, “Blobworld: A System forRegion-Based Image Indexing and Retrieval”, ThirdInt. Conf. On Visual Information and InformationSystems, Amsterdam, 1999, pp. 509-516.[5] A. Hampapur, A. Gupta, B. Horowitz, C-F. Shu,C. Fuller, J. Bach, M. Gorkani, R. Jain, “Virage VideoEngine”, SPIE Vol. 3022, 1997.[6] D. Ponceleon, S. Srinivasan, A. Amir, D.Petkovic, D. Diklic, “Key to Effective VideoRetrieval: Effevtive Cataloging and Browsing”, ACMMultimedia, ’98, pp. 99-107.[7] S-F. Chang, W. Chen, H. Meng, H. Sundaram,D. Zhong, “A Fully Automated Content Based VideoSearch Engine Supporting Spatio-Temporal Queries”,IEEE Transaction on Circuits and Systems for VideoTechnology, Vol. 8, No. 5, Sept., 1998.[8] T. G. Aguierre Smith, G. Davenport, TheStratification System: A Design Environment forRandom Access Video, Workshop on Network andoperating system, La Jolla, CA, 1992.[9] R. Weiss, A. Duda, D. K. Gifford, Content-basedAccess to Algebraic Video, Int. Conf. on MultimediaComputing and Systems, IEEE Press, 140-151.[10] E. Oomoto, K. Tanaka, OVID: Design andImplementation of a Video-Object Database System,IEEE Trans Knowl Data Eng, 5(4), 1993, 629-643.[11] S. Adali, K. S. Candan, S-S. Chen, K. Erol, V. S.Subrahmanian, “Advanced Video InformationSystem: Data Structure and Query Processing”,Multimedia System Vol. 4, No. 4, Aug. 1996, pp.172-86.[12] C. Decleir, M-S. Hacid, J. Kouloumdjian, “ADatabase Approach for Modelling and QueryingVideo data”, LTCS-Report 99-03, 1999.[13] H. Jiang, A. Elmagarmid, “Spatial and temporalcontent-based access to hypervideo databases” VLDBJournal, 1998, No. 7, pp. 226-238.[14] J. Z. Li, M. T. Ozsu, D. Szafron, “Modeling ofVideo Spatial Relationships in an Object DatabaseManagement System”, Proc. of Int. Workshop onMulti-media Database Management Systems, 1996,pp. 124-132.[15] A. Woudstra, D.D. Velthausz, H.J.G. de Poot, F.Moelaert El-Hadidy, W. Jonker, M.A.W. Houtsma,R.G. Heller, J.N.H. Heemskerk, Modelling andRetrieving Audiovisual Information - A Soccer VideoRetrieval System -, 4th International Workshop onMultimedia Information Systems; Istanbul, Turkey,September 1998.[16] D. Zhong, S-F.Chang, Video Object Model andSegmentation for Content-Based Video Indexing,IEEE Circuits and Systems, Hong Kong 1997.[17] Y. Gong, L. T. Sin, C. H. Chuan, H-J. Zhang, M.Sakauchi, “Automatic Parsing of TV SoccerPrograms”, IEEE International Conference onMultimedia Computing and Systems, Washington D.C., 1995, pp. 167-174.[18] S. Intille, A. Bobick, “Visual Tracking UsingClosed-Worlds”, M.I.T. Media Laboratory, TechnicalReport No. 294, Nov. 1994.[19] G. P. Pingali, Y. Jean I. Carlbom,“LucentVision: A System for Enhanced SportsViewing”, Proc. of Visual’99, Amsterdam, 1999, pp.689-696.[20] M. Petkovic, W. Jonker, “A Framework forVideo Modelling”, 18th IASTED Conference onApplied Informatics, Innsbruck, Austria, 2000.[21] D. Papadias, Y. Theodoridis, T. Sellis, and M.Egenhofer , “Topological Relations in the World ofMinimum Bounding Rectangles: A Study with R-Trees”, SIGMOD ’95, San Jose, CA, M. Carey and D.Schneider (eds.), SIGMOD RECORD 24 (2): 92-103,May 1995.[22] J. F. Allen, Maintaining knowledge abouttemporal intervals, Communications of ACM, 26(11),1983, 832-843.[23] P. Boncz, A.N. Wilschut, M.L. Kersten,“Flattering an object algebra to provide performance”,Proceedings of the 14th IEEE International Conferenceon Data Engineering, Orlando, Florida, 1998, pp. 568-577.[24] P. Boncz, M.L. Kersten, “Monet: AnImpressionist Sketch of an Advanced DatabaseSystem”, Proceedings Basque International Workshopon Information Technology, San Sebastian, Spain,July 1995.