In automatic video content analysis domain, the key challenges are how to recognize important objects and how to model the spatiotemporal relationships between them. In this paper we propose a semantic content analysis model based on Perception Concepts (PCs) and Finite State Machines (FSMs) to automatically describe and detect significant semantic content within sports video. PCs are defined to represent important semantic patterns for sports videos based on identifiable feature elements. PC-FSM models are designed to describe spatiotemporal relationships between PCs. And graph matching method is used to detect high-level semantic automatically. A particular strength of this approach is that users are able to design their own highlights and transfer the detection problem into a graph matching problem. Experimental results are used to illustrate the potential of this approac

Bai, Liang

Jones, Gareth J.F.

Lao, Songyang

Smeaton, Alan F.

English

Irish Universities

A semantic content analysis model for sports video based on perception concepts and finite state machines

Crossref

A Semantic Content Analysis Model for Sports Video Based on Perception Concepts and Finite State Machines

DCU Online Research Access Service

A SEMANTIC CONTENT ANALYSIS MODEL FOR SPORTS VIDEOBASED ON PERCEPTION CONCEPTS AND FINITE STATE MACHINESLiang Bail 2, Songyang LaoI, Gareth J.F.Jones2, Alan F.Smeaton2School of Information System & Management, National University ofDefense Technology,ChangSha, China, 410073lbaigcomputing.dcu.ie, laosongyanggvip.sina.com2Centre for Digital Video Processing, Dublin City University, Glasnevin, Dublin 9, Ireland{gjones, asmeaton}lcomputing.dcu.ieABSTRACTIn automatic video content analysis domain, the keychallenges are how to recognize important objects and howto model the spatiotemporal relationships between them. Inthis paper we propose a semantic content analysis modelbased on Perception Concepts (PCs) and Finite StateMachines (FSMs) to automatically describe and detectsignificant semantic content within sports video. PCs aredefined to represent important semantic patterns for sportsvideos based on identifiable feature elements. PC-FSMmodels are designed to describe spatiotemporal relationshipsbetween PCs. And graph matching method is used to detecthigh-level semantic automatically. A particular strength ofthis approach is that users are able to design their ownhighlights and transfer the detection problem into a graphmatching problem. Experimental results are used toillustrate the potential of this approach.1. INTRODUCTIONNowadays we can access increasing amounts of video dataas a result of the development of high-speed broadbandnetworks and digital video technology. One of the areas ofgreatest expansion in video content is sports broadcasting.An important component of sports broadcasting is highlightsof sports games which are usually prepared manually.Preparing this is time-consuming and the situation isinflexible with respect to individual viewers who may wanta longer or shorter summary or to focus on certain eventtypes. Further the need for manual editing means that oftenthe generation of any summary at all is not cost effective.It is important for video processing and retrieval researchersto develop new ways to model interesting semantic contentand identify semantic events automatically in sports videos.To date work in this area has focused on query-by-text. Theuser enters the word "goal", then the system searches forvideo clips including the word goal in the soundtrack orpossibly in manually added metadata [1][2]. This clearlyrelies on either a well annotated soundtrack or a costlymanual labeling of semantic events. Automatic discovery ofsemantic content that captures the essential contents of agame is becoming more and more important. Related priorwork towards automatic event detection in sports videos isdescribed in [3][4][5] and[6]. In most existing work theevent detection algorithms are embedded in systems andcannot easily be redefined. This means that users cannotadapt the event types detected or the system refined to thedifferent editorial rules used by different broadcastingcorporations. In our work, we have proposed a semanticdescription method based on Petri-Net in [7] and used FiniteState Machines to detect semantic events in soccer video [8].In this paper we introduce a semantic content analysismodel for sports video. For this model we first define PCs todescribe patterns sharing similar spatiotemporal behaviors insports video. PCs are combined into FSMs which describethe spatiotemporal relations between the PCs associatedwith significant semantic content in a game. PC-FSMs aredescribed formally in terms of state graphs. A graphmatching method is used to discover semantic contentautomatically. Finally we illustrate the validity of this modelusing experiments on recorded sports videos.2. PERCEPTION CONCEPTS IN SPORTS VIDEOIn the sports domain, a TV sports program editor isinterested in selecting similar and periodic actions whichcan help the audience to understand and enjoy a game. Inthis case, it is important that the similar and periodic-actionpatterns share similar spatiotemporal behaviors that can beclustered and described with a linguistic concept. Theserequirements motivate the possibility that patterns that sharethe same behaviors can be represented by PCs. PCs in sportsvideo are abstractions of video elements and can be of twomain types: Visual Concepts and Aural Concepts. In thissection we outline the characteristics ofPC types.* Visual ConceptsVisual concepts in sports videos share the same visualfeatures. Visual concepts can be of different types:sequence, object and slow-motion-replay.1-4244-1017-7/07/$25.00 C2007 IEEE 1407 ICME 2007Sequence: a sequence of frames captured by one camera ina single continuous action with fixed focus and steadycontent.Object: a segmentable region representing a distinctsemantic concept in a frame.Slow-motion-replay (SMR): is a sequence of framesplayed in a low play rate to replay video content of anearlier event, which is a special and important component ofsports video.* Aural ConceptAn aural concept is a temporal segmentation of the audiotrack and represents a distinct semantic audio type. Auralconcepts are generated by audio segmentation andclassification. Aural concepts, such as: cheers from theaudience and speech from commentators, are useful forsemantic analysis of games.3. DEFINITION OF PC-FSM MODELA FSM is an abstract machine consisting of a set of states, aset of input events, a set of output events, and a statetransition function. The function takes the current state andan input event, and returns the new set of output events andthe next state. FSMs are effective for modeling sequentialprocesses. Formally, a PC-FSM is defined as follows:Definition 1 A PC-FSM is a 7-tupleCPC_FSM = {SPC, SO, I, O, T, Op, Dt}where: SPC is the set of states in sports video.SPC= {JIF NV, OF NV,TV,SMR}The elements in Spc respectively represent Infield NormalView, Out offield Normal View, Tight View, Slow MotionReplay.S0 represents the initial state.I is the input event set. For a PC-FSM model, this drives thetransitions of PC states. For a strict description, Nulldescribes transition without any input events and Event-Endindicates the end of a semantic event.O is the output set. When an I event happens, someperception concepts occur, such as a special object.T is a finite set of transitions. Each of the transitions t in Tcan be defined as : t: (Head(t), I(t),O(t, op), Tail(t))where, Head (t) is the starting state; I (t) is the input event oft; 0 (t, op) is the output event set of t; op indicates the logicrelation between output events and t, it can be figured as:Event op, op e Op. Tail(t) is the end state.Op is a set of logic operators that indicate the logic ofrelations among events and between events and transitions.For the PC-FSM model we define Op set as follow:Op = {following, before, synchronazition}Video is different to text since there are temporal relationsbetween events and transition. So we define following,before and synchronization as tokens to describe thesequence of output events and transitions.Dt represents the duration time of a state or an output event.Definition 2 Dt is a 2-tupleDt(Operator, Time).where: Operator {<, =, >}, Time E R .Dt is an important parameter in the query processing. Forexample, in the semantic of Goal, the duration of Tight Viewstate is very long.In PC-FSM model, the set of output events 0 is as follows:0 = {Null, Visual Objects, Aural Concepts}The Null element represents no output events occurringduring a transition.A PC-FSM can be represented by a directed graph G(V, E)institutively, where V, the vertexes set, represents the statesof PC-FSM and E, the edges set, represents the transitionsbetween different states.4. SEMANTIC CONTENT DESCRIPTION ANDDISCOVERING BASED ON PC-FSM MODELUsually, a sports match is composed of In-Play and Out-of-Play. Out-of-Play events are always brought about byspecial events, such as a foul in soccer or basketball. An In-Play scene is composed of a mass of LVs and MVs with theplaying field as background and a spot of TVs. An Out-of-Play scene is composed of TVs, LVs with auditoria asbackground and MSRs inserted by the editor. Events insports matchs often occur in In-Play and drive a sequence ofOut-of-Play events, including TV interviews with relevantpeople, such as players or referees, and MSRs for replay forthe events and so on. This structure of sports video contentcan be described as in Figure 1.Sports video-l.....--.-.-.1 ...... ....*:::... Out of Play ...::*:*:*L::: YTV:: <T:: LV MSREvent ...Figure 1. Structure of sports video contentThis structure can be studied from a training video data setand defined by a PC-FSM model according to the user'sknowledge of a sports domain. Then we can automaticallydiscover semantic content using defined PC-FSM models.We take popular sports, soccer and basketball, as examplesfor the application of the PC-FSM model.4.1 Semantic Content DescriptionAccording to observations for sports video, sequence visualconcepts can be classified as: Out-of-Field View (OFV), In-Field Loose View (IF-LV), In-Field Medium View (IF-MV)and Tight View (TV) (see Figure 2). Dominant color featurewhich represents local features, where a small number ofcolors are enough to characterize the color information inthe region of interest (for example green for soccer field andyellow for basketball in Figure 2), can be used for sequence1408classification [9]. The IF-LV and IF-MV share analogicalvisual features and are often associated with a one-shotzoom action. They can be defined as one visual conceptstyle named In-Field Normal View (IN-NV), as adopted inthis paper.Loose View Media View Tight View OFVFigure 2 Sequences in Sports Game.Only a limited number of object types are observed insports videos, such as: player, referee, coach, captions andso on. Here we only select one object: caption, which can bedetected reliably in sports video [9], and is thus available tobe described as semantic content. In general, in a sportsmatch there are two kinds of important audio: whistles andcheers. High crowd noise with low or absent speech is oftenassociated with loud cheers. A whistle from a referee hashigh frequency and a strong spectrum [10]. It can bedetected according to peak frequencies which fall within thethreshold range. A whistle sound is useful to suggest thatsomething interesting has happened. So we consider whistleand cheers as aural concept here. We describe Goal Scoredin soccer and Foul in basketball using PC-FSM model. S0 isset to IN-NV because when an event begins and ends thecamera view mainly focuses on the field with an IN-NV.* Goal Scored in soccerThe PC-FSM of Goal Scored is shown in Figure 3.t(, 20s)X N , Dt V) \ - )1) XNVFigure 3. PC-FSM Graph of Goal ScoredtO: KIF - NV, Goal Scored, {Whistle before, Cheer synchronization}, TV)tl: (TV, Event - End,{Caption ffollowing}, IN - NV)t2: (TV, Null,{Cheer synchronization},OF - NV)t3: (OF - NV, Null,{Cheer synchronization}, TV)t4: (TV, Null,{Cheer synchronization}, MSR)t5: (MSR, Null, {Cheer synchronization}, TV)t6: (OF - NV, Null, {Cheer synchronization}, MSR)t7: (MSR, Event - End,{Caption ffollowing}, IF - NV)* Foul in basketballThe PC-FSM of Foul is shown in Figure 4.t4t3 t2MSRto ~~~~~tiFigure 4. PC-FSM Graph of FoulTransitions in Figure 4 are defined as follow:tO: (IF - NV,Foul,{ Whistle before},TV)tl: (TV, Null, Null,MSR)t2: (MSR,Null,Null,TV)t3: (TV, Event - End, Null, IN - NV)t4: (MSR,Event - End,Null,IN - NV)Other semantic content can be similarly described in thesame way using the PC-FSM model. A particular strength ofthis approach is that the user can modify or define new PC-FSMs to describe different semantic content based on theirknowledge of activities in the sports domain.4.2 Semantic Content DiscoveringBased on the PC-FSM model, automatic discovery ofsemantic content in sports video can be designed. Theexperiments described here used a manually annotateddatabase of video shots. This was used in order to eliminateeffects of errors in shot cluster and object recognition, sincein this paper we focus on demonstrating the validity of thePC-FSM model. After manual annotation the video of amatch is converted to a series of states with output PCs.Semantic content discovery is carried out in two steps:Firstly, candidate events are detected in the sequence ofvideo states. According to the definition of the PC-FSMmodel, the initial state is IF-NV. So the sports video issegmented by occurrence of the IF-NV state. The sequencebetween one IF-NV and the next is considered as acandidate event.The second step is matching between candidate events andthe PC-FSM model of semantic content. The candidateevent can be described formally as a state graph with sometransitions like PC-FSMs graph. Then event detection canbe carried out using a graph matching method.Definition 3 For PC-FSM graphs Gc and Gb,if Sa = Sb , Dt, = Dtb, for each S and transitionSeta ( transitionSetbthen Ga 5 GbThe rule for semantic event detection is: if the PC-FSMgraph for a candidate event belongs to a given PC-FSMgraph, a semantic event is detected and annotated.5. EXPERIMENTS AND EVALUATIONIn order to demonstrate our approach to identifying semanticcontent in sports video we conducted a preliminary set ofexperiments. These were carried out using five soccergames and three basketball games recorded from 4:2:2 YUVPAL tapes as MPEG-I format. The soccer videos are fromtwo UK broadcasters (ITV and BBC Sport), and are taken1409from the 2006 World Cup, taking a total of 7hs 53mins 28s.And the basketball videos are NBA games recorded fromESPN, FOX Sports and CCTV5 taking a total of 6hs 47mins18s.For soccer videos we defined Goal Scored (GS), Foul (S-F)and Yellow (or Red) Card (YRC) events and detect thembased on PC-FSM model. Highlight Attack (HA) and Foul(B-F) events are defined and detected in basketball videos.Table 1 shows "Precision" and "Recall" for detection of thesemantic events. The ground truth is recognized manually. Itcan be seen that the precision and recall are higher than 87%.Table 1. Precision and recall for sports semanticssemantic GS S-F YRC HA B-FPre (%) 100 89.9 92.9 88.4 87.7Rec(%) 100 87.6 100 93.2 96.5We also compared our approach with other approaches. Inour previous work, a Petri-Net (PN) model is used for videosemantic content description and detection [7]. 11MM is apopular model for video event detection [5][9]. In ourexperiments, we use the PN based approach and HMMbased approach proposed in [9] to detect semantic contentusing same video data set. The results are shown in Table 2.From Table 2, we can find the precision and recall of PNbased approach is almost equivalent with PC-FSM basedapproach. But the key problem for PN based approach isdifficult to transfer a complex PN model to SQL queryautomatically. PN model is stronger in semantic contentdescription but weaker in detection processing.Low precision and recall are shown in the experimentalresults of HIMM based approach, in which low-level features,color, texture and motion vector, are extracted to trainingdifferent HMM models for different semantic content. Thisapproach tends to map low-level features to high-levelsemantic directly according to statistic point, which cancapture perception feature pattern well but not be effectiveto model and detect spatiotemporal relationship betweendifferent semantic content.Table 2. Results based on PN and HMM Approachsemantic GS S-F YRC HA B-FPN Pre(%) 85.2 86.6 91.7 85.8 84.5Rec(%) 100 84.1 97.5 91.6 90.3HMM Pre(%) 75.4 63.8 77.6 61.5 59.2Rec(%) 80.1 72.5 83.1 64.9 67.3The PC-FSM based approach defines PCs as middle-levelsemantic content that represents perception patterns andnarrows the gap between low-level features and high-levelsemantic content. Using a graph match method fordiscovering high-level semantic content avoids the difficultyof generating complex SQL queries. Based on the aboveexperimental results, we believe that our approach tosearching in sports video has considerable potential.proposed a semantic content analysis model using FSMs.The effectiveness of this model was demonstrated throughpreliminary experiments. Our method can be utilized fordifferent sports video. Future work will explore interfacedesign, which is very important to enable rapid developmentof descriptions for the new event types, and researchintelligent methods for PC-FSM model graph matching.7. ACKNOWLEDGMENTSThis work is supported by the National High TechnologyDevelopment 863 Program of China (2006AAOlZ316), theNational Natural Science Foundation of China (60572137)and China Scholarship Council of China Education Ministry.8. REFERENCES[1] D. Zhang and D. Ellis, "Detecting Sound Events in BasketballVideo Archive", Technical Report, Dept. of Electrical Engineering,Columbia University, 2001.[2] R. Dahyot, A. C. Kokaram, N. Rea and H. Denman, "JointAudio-Visual Retrieval for Tennis Broadcasts", In Proceedings ofthe 28th IEEE International Conference on Acoustics, Speech andSignal Processing (ICASSP'03), Hong Kong, April 2003.[3] D. Zhang, S-F. Chang, "Event Detection in Baseball VideoUsing Superimposed Caption Recognition", In Proceedings of the10th ACM International Conference on Multimedia, pp315-318,Juan-les-Pins, France, Dec 2002.[4] R.Leonardi and P. Migliorati, "Semantic Indexing ofMultimedia Documents", IEEE MultiMedia, Vol.9, No. 2, pp44-51, April/June 2003.[5] Guoying Jin, Linmi Tao, Guangyou Xu, "Hidden MarkovModel Based Events Detection in Soccer Video", InternationalConference of Image Analysis and Recognition, Porto, Portugal,October 2004, LNCS 3211, pp. 605-612[6] D.A.Sadlier and N.E. O'Connor, "Event Detection in FieldSports Video Using Audio-visual Features and A SVM", WEEETransactions on Circuits and Systems for Video Technology,15(10), 1225-1233, 2005.[7] S.Y. Lao, A. F. Smeaton, G. J. F. Jones, H. Lee, "A QueryDescription Model Based on Basic Semantic Unit Composite Petri-Nets for Soccer Video Analysis", In Proceedings ofACM MIR'04,October 15-16, 2004, New York, USA[8] L. Bai, S.Y. Lao, W.M. Zhang, A. F. Smeaton, G. J. F. Jones,"A Semantic Event Detection Approach for Soccer Video based onPerception Concepts and Finite State Machines", InternationalWorkshop on Image Analysis for Multimedia Interactive Services,Santorini, Greece, 6-8 June 2007.[9] J.Y. Chen, Y.H. Li, S.Y. Lao, et al, "Detection of ScoringEvent in Soccer Video for Highlight Generation", TechnicalReport, National University of Defense Technology, 2004.[10] Zhou, W., S. Dao, and C.-C. Jay Kuo, "On-line Knowledgeand Rule-based Video Classification System for Video Indexingand Dissemination", Information Systems, 2002. 27(8): p. 559-586.6. CONCLUSIONS AND DISCUSSIONSIn this paper, based on analyzing sports videocharacteristics, we define PCs for sports games and1410

A Query Description Model Based on Basic Semantic Unit Composite PetriNets for Soccer Video Analysis&quot;,

A Semantic Event Detection Approach for Soccer Video based on Perception Concepts and Finite State Machines&quot;, International Workshop on Image Analysis for Multimedia Interactive Services,

Detecting Sound Events in Basketball Video Archive&quot;,

Detection of Scoring Event in Soccer Video for Highlight Generation&quot;,

Event Detection in Baseball Video Using Superimposed Caption Recognition&quot;,

Event Detection in Field Sports Video Using Audio-visual Features and A SVM&quot;,

Hidden Markov Model Based Events Detection in Soccer Video&quot;,

Joint Audio-Visual Retrieval for Tennis Broadcasts&quot;,

On-line Knowledge and Rule-based Video Classification System for Video Indexing and Dissemination&quot;, Information Systems,

Semantic Indexing of Multimedia Documents&quot;,

Name not available

http://doras.dcu.ie/220/1/ieee_icme_2007.pdf

A semantic content analysis model for sports video based on perception concepts and finite state machines

Abstract

Similar works

Full text

Available Versions

Irish Universities

Crossref

DCU Online Research Access Service

Name not available