In this paper we present different sources of information complementary to audio-visual (A/V) streams and propose their usage for enriching A/V data with semantic concepts in order to bridge the gap between low-level video analysis and high-level analysis. Our aim is to extract cross-media feature descriptors from semantically enriched and aligned resources so as to detect finer-grained events in video.

We introduce an architecture for complementary resources analysis and discuss domain dependency aspects of this approach connected to our initial domain of soccer broadcasts

Buitelaar, Paul

Cobet, Andreas

Declerck, Thierry

Nemrava, Jan

O'Connor, Noel E.

Petrák, Josef

Sadlier, David A.

Svátek, Vojtech

Zeiner, Herwig

In this paper we present different sources of information complementary to audio-visual (A/V) streams and propose their usage for enriching A/V data with semantic concepts in order to bridge the gap between low-level video analysis and high-level analysis. Our aim is to extract cross-media feature descriptors from semantically enriched and aligned resources so as to detect finer-grained events in video.\ud
We introduce an architecture for complementary resources analysis and discuss domain dependency aspects of this approach connected to our initial domain of soccer broadcasts

Name not available

Abstract— In this paper we present different sources ofinformation complementary to audio-visual (A/V) streams andpropose their usage for enriching A/V data with semanticconcepts in order to bridge the gap between low-level videoanalysis and high-level analysis. Our aim is to extract cross-media feature descriptors from semantically enriched andaligned resources so as to detect finer-grained events in video.We introduce an architecture for complementary resourcesanalysis and discuss domain dependency aspects of thisapproach connected to our initial domain of soccer broadcasts.Index Terms— Multimedia databases, Text processingI. INTRODUCTIONCurrent progress in technology together with changinglifestyle towards mobile devices and on-demand videodelivery allows a user to view video content almost anywhereat any time. However people are often interested only in somekind of highlight events within a larger volume of audiovisualdata. This counts for almost all video genres available.Despite the advances in content-based video analysistechniques, the quality of video retrieval would stronglybenefit from exploitation of related (complementary) textualresources, especially if these are endowed with temporalreferences. Good examples can be found within the sportsdomain [1]. Current research in sports video analysis focuseson event recognition and classification based on theextraction of low-level features and is limited to a very smallnumber of different event-types, e.g. ‘scoring-event’. On theother hand, vast textual data sources can serve as a valuablesource for finer-grained event recognition and classification.In particular, textual data can be exploited as backgroundknowledge in filtering the video analysis results and thusimprove the corresponding algorithms.We introduce a generic architecture for complementaryresource exploitation, and discuss domain dependency issues.We further describe concrete sources of complementaryinformation that we came across in our application domain(soccer), such as real-time game logs (minute-by-minutereports), textual summaries found on websites, OCR on thevideo content, speech transcripts and others, and we describethe ways they can be merged together to create a coherenttextual match description. We also relate them to the corevideo analysis framework.II. PROPOSED ARCHITECTURE AND RESOURCESFigure 1 shows the proposed framework [3]. The process ofgathering, alignment and mapping of the complementary datato video can be divided into five different phases:1. a) Gathering and preprocessing the textual sources,b) Building the database of video analysis results2. Mutual synchronization of the textual sources and buildingdatabase of ontological resources.3. Alignment of video data, textual data and knowledge db.4. a) Time synchronization of textual and video data.b) Event type recognition5. Annotation results provision and cross-media featureextraction. The former directly link concrete events in videoto semantic categories. The latter characterize semanticcategories in terms of typical values of video-orienteddescriptors.Figure 1. Proposed ArchitectureTextual resources are the main semantics carrier withinthis architecture. (Semi-)Structured, database-like textualresources contain the summary of larger-scale events coveredby individual video broadcasts (such as soccer matches), and,thanks to their clean structure, provide valuable and easilyextractable information. On the other hand, unstructuredevent reports require more sophisticated techniques; forexample, they can be extracted from web sites using wrappersand then information extraction tool [5] can be applied inorder to extract domain-related ontology concepts.Minute-by-minute reports are an example of free textinformation structured by the time points of finer-grainedevents not covered by structured ‘protocols’. CombiningJ. Nemrava(1,2), P. Buitelaar(2), T. Declerck(2), V. Svátek(1), J. Petrák(1), A. Cobet(3), H. Zeiner(4), D. Sadlier(5), N. O’Connor(5)Architecture for Enhancing Video AnalysisResults using Complementary Resources1) Univ. of Economics, Prague, CZ 2) DFKI Saarbruecken, DE 3) TU Berlin, DE, 4) Joanneum Research, Graz, AT 5) Dublin City Univ., IR7several of these reports [7] can increase the probability ofcovering unidentified highlights detected by the videoanalysis.While web-based reports and semi-structured ‘databases’ canbe viewed as ‘secondary’ complementary resources, as theyare not directly connected to the video, OCR on text presentin videos in the form of overlays represents a typical exampleof ‘primary’ complementary resources. Temporal alignmentis not needed here. General text detection in video is morecomplex, because text may appear for example on signs (shopname, city name, street names…), on non-rigid objects (e.g.T-Shirt of a person with text) and so on. The OCR tests weredone on key frames from soccer matches. This promisingsource of information will provide us with more textualinformation about what is actually happening in the game and– what’s most important – provides a way to synchronizevideo file time with the actual time of the match, using thescoreboard analysis.A sequence of 16 frames is used for detection of text as itcarries information about moving objects and static text. Forexample in a soccer game the camera is moving all the timeor the players are moving, but the text with the score, time,and teams are always on the same place in the frames. Thusmost of the moving objects are removed and only the staticedges of the text areas are left. In this way it is possible todetect the text regions as shown below on Figure 2.A different kind of ‘primary’ complementary resources arespeech transcripts. Their analysis on the soccer domain isdescribed in the MUMIS project [4]. The baseline automaticspeech recognition system yields Word Error Rates (WERs)that varied from 84% to 94% for the different languages andtest matches. Application specific words such as players’names are recognized correctly in about 50% of cases. Thereason why the WERs are so high is because of the extremelyhigh level of stadium noise present in the audio track.A domain knowledge base serves as an ontologicalresource. In our soccer domain, it contains information suchas player database (a list of players names, their numbers,substitutions etc), match metadata (basic information aboutthe game such as date, place, referee name or attendance) andevent database (list of possible event types and their cross-media feature descriptions).The scope of video analysis tools is very broad and differsfrom one domain to another. In our experiments, we reliedon low-level feature extraction from soccer matches as relyingon six generic sports-video detectors (i.e. crowd/spectatordetection, audio energy envelope, close-up detection,scoreboard activity measure, motion activity quantification,field-line extraction). More details may be found in [2].III. DOMAIN DEPENDENCYA crucial question is whether our experiments are to somedegree reusable in different domains and settings than theanalysis of soccer videos. The reusability can roughly beconsidered at several levels:1. Different categories of soccer (soccer) matches and/ordifferent styles of audio/video data recording2. Different groups of sports, which can be determinedaccording to multiple facetsa. field sports vs. othersb. temporally structured (and in gross time vs. net timewhere breaks and pauses are not included) vs. thosestructured by score (or similar non-temporal aspect)c. collective vs. individual sports.3. Beyond the sports domain.When looking at non-sports events, the applicability of ourapproach is strongly limited by the unavailability of temporalonline reports in many cases as well as limited possibility tovisualise events in more ‘spiritual’ domains such as politics.IV. CONCLUSIONSWe presented an architecture for complementary resourcemining and for their mapping to video analysis results, whichalso yields cross-media descriptors. The research has asstarting point our experiments in the soccer domain. In ourfuture research we will to focus on creation of sports-genre-specific detectors (e.g. for corner flag) and compare theperformance of adding extra detector with the performancewhen using just general detectors. We want to test a videodetector based on specific event-type described in the textualdata. We believe that this will help us better assess andpossibly overcome the domain dependency of our approach.ACKNOWLEDGMENTThis research was supported by the European Commissionunder contract FP6-027026 for the K-Space project.REFERENCES[1] Xu, H. and Chua, T. 2004. The fusion of audio-visual features andexternal knowledge for event detection in team sports video. In Proceedingsof the 6th ACM SIGMM Workshop on Multimedia information Retrieval[2] Sadlier D. and O'Connor N.: Event Detection in Field Sports Video usingAudio-Visual Features and a Support Vector Machine. IEEE Transactionson Circuits and Systems for Video Technology, Oct 2005[3] Nemrava et al.: Architecture for mapping between results of video analysisand complementary resource analysis., K-Space Public Deliverable 5.10[4] Sturm J. et al.: Automatic Transcription of Football Commentaries in theMUMIS Project. In EUROSPEECH-2003, p 1853-1856.[5] Drozdzynski W., Krieger H.-U., Piskorski J., Schäfer U., Xu F.. ShallowProcessing with Unification and Typed Feature Structures - Foundationsand Applications. In Künstliche Intelligenz 1/2004.[6] Arndt, R. et al.: Architecture Specification of the K-Space AnnotationTool, K-Space Deliverable D5.11. Public[7] Nemrava J., Buitelaar P., Svátek V., Declerck T.: Event Alignment ForCross-Media Feature Extraction In The Football Domain. WIAMISSantorini : IEEE Computer Society, 2007, s. 1–3. ISBN 0-7695-2818-X.Figure 2. Fusion of frames for moving object detection8

Architecture for enhancing video analysis results using complementary resources

In this paper we present different sources of information complementary to audio-visual (A/V) streams and propose their usage for enriching A/V data with semantic concepts in order to bridge the gap between low-level video analysis and high-level analysis. Our aim is to extract cross-media feature descriptors from semantically enriched and aligned resources so as to detect finer-grained events in video.
We introduce an architecture for complementary resources analysis and discuss domain dependency aspects of this approach connected to our initial domain of soccer broadcasts

Architecture for enhancing video analysis results using complementary resources

Abstract

Similar works

Full text

Available Versions

Name not available

Irish Universities

DCU Online Research Access Service