The paper presents an approach to segmenting broadcast TV news programmes automatically into individual news stories. We first segment the programme into individual shots, and then a number of analysis tools are run on the programme to extract features to represent each shot. The results of these feature extraction tools are then combined using a support vector machine trained to detect anchorperson shots. A news broadcast can then be segmented into individual stories based on the location of the anchorperson shots within the programme. We use one generic system to segment programmes from two different broadcasters, illustrating the robustness of our feature extraction process to the production styles of different broadcasters

Czirjék, Csaba

Murphy, Noel

O'Connor, Noel E.

O'Hare, Neil

Smeaton, Alan F.

English

DCU Online Research Access Service

A GENERIC NEWS STORY SEGMENTATION SYSTEM AND ITS EVALUATIONNeil O'Hare, Alan F. Smeaton, Csaba Czirjek, Noel O'Connor, Noel MurphyCentre for Digital Video Processing, Dublin City University, Glasnevin, Dublin 9, Irelandnohare@computing.dcu.ieABSTRACTThis paper presents an approach to automaticallysegmenting broadcast TV news programmes intoindividual news stories. We first segment the programmeinto individual shots, and then a number of analysis toolsare run on the programme to extract features to representeach shot. The results of these feature extraction tools arethen combined using a Support Vector Machine trained todetect anchorperson shots. A news broadcast can then besegmented into individual stories based on the location ofthe anchorperson shots within the programme. In thispaper we use one generic system to segment programmesfrom two different broadcasters, illustrating the robustnessof our feature extraction process to the production stylesof different broadcasters.1. INTRODUCTIONOver the last decade the problems involved with thecapture, compression, streaming, transmission andplayback of digital video information have largely beensolved. This has meant that it is now possible to deploydigital video on a large scale. Once video contentbecomes available in very large quantities, however, itbecomes important to be able to access this content asefficiently as possible.This situation has led to much research in the areas ofmultimedia analysis and retrieval. The Centre for DigitalVideo Processing (CDVP) at Dublin City University hasbeen pursuing an ongoing research programme to developtechnologies for the efficient management of digital videocontent. The work of the Centre is demonstrated via theweb-based Físchlár systems, which include the dedicatedTV news sub-system Físchlár-News [1].There has been much work in video processing in thearea of temporal segmentation since an important task inmanaging digital assets is to break them down intomanageable units. Often this segmentation is at the levelof the shot, the task being to detect the boundariesbetween different camera set-ups. A summary of the videocan then be presented based on this segmentation [2]. Thisshot-level segmentation can be understood as thedetection of syntactic boundaries within digital videofiles, but in many cases a semantic segmentation, wherehigher level scene or story level changes are detected, ismore useful. Detecting semantic boundaries is a difficulttask, but in the case of news broadcasts we can use apriori knowledge of the structure of the broadcast to guideus in our segmentation (eg [3]).In recognition of the fact that news broadcasts are animportant genre, and their segmentation an important task,a Story Segmentation task was included as part ofTRECVID2003 [4].In this paper, we outline our own work in the StorySegmentation task for TRECVID2003. In the next sectionwe outline the motivation for our approach and give anoverview of our system.  Section 3 describes the analysistools we run on news broadcasts to extract features to beused for anchorperson detection. Section 4 describes howthese features are combined to detect the anchorpersonshots in a news broadcast using a Support VectorMachine. In Section 5 some results are presented anddiscussed, and in Section 6 some conclusions anddirections for our future research are outlined.2.  SYSTEM OVERVIEWIn our work on News Story Segmentation, we exploit thestructure of news broadcasts in order to detect storyboundaries. Each story within a broadcast usually beginswith a leading shot of the anchorperson introducing thenew story. This anchor shot is usually followed by areport with more details on the story.  Anchor shots aretypically filmed in a studio location, and within a singlebroadcast they are captured via the same camera set-up,and as such exhibit strong visual similarity. This structureis illustrated in Figure 1.?????????????????????????????????? ????????? ???????????Anchorperson ShotsCommercial BreakNews Report ShotsFigure 1. Structure of a News BroadcastOur approach, outlined in Figure 2, is to detect theanchorperson shots by combining the results of a numberof automatic video analysis tools: we make theassumption that all anchorperson shots signal a new storyso we log stories at the beginning of anchorperson shots.We first segment the broadcast into shots, and subsequentanalysis takes place at the level of the shot. We run anumber of analysis tools to extract high-level featuresdescribing each shot in the video sequence, and then wecombine these features using a trained Support VectorMachine (SVM).  The resulting trained SVM classifiercan be used to detect the anchorperson shots within abroadcast.Shot ClusteringFace DetectionMotion ActivityAnalysi s  2xShot LengthSupport VectorMachineNews StoryDetectionNews StoriesDonated by CLIPS-IMAG30 Minute News ProgramShot BoundaryDetectionShot Level Feature ExtractionFigure 2. Overview of News Story Segmentation system.3. VISUAL ANALYSIS FOR PRE-PROCESSINGAND FEATURE EXTRACTIONThe first stage of our news story segmentation approach isto segment each news broadcast into individual shots. Forthe TRECVID2003 story segmentation task, we used thecommon shot boundaries provided by the CLIPS-IMAGgroup [4] and used by all TRECVID participants. We thenanalyze each shot in the broadcast to extract a number offeatures representing the shot, as described below.3.1. Shot ClusteringWe have previously developed an approach for detectinganchorperson shots in a news programme based on shotclustering [5].This method works by grouping together shots that arevisually similar to each other, based on colour histograms.The Cosine similarity measure is used to calculate adistance between the keyframes representing each shot,and shots are clustered based on this distance.Since the anchorperson shots in a TV news broadcastare very similar to each other, they are placed in the samecluster by this process.  We can then apply someheuristics to distinguish the anchorperson groups from thenon-anchorperson groups:? The temporal range of the shots must be higher than apre-determined threshold. Anchorperson shots tend tobe dispersed throughout a broadcast, so shots that arevery similar visually but occur only very closetogether in the broadcast should be rejected.? The group similarity mean should be higher than avery high threshold.  This is because anchorpersonshots are extremely similar to each other and form‘tight’ clusters.? The mean anchorperson shot length should be longerthan a minimum threshold. This is becauseanchorperson shots are generally quite long incomparison with other video content, rarely lastingless than five seconds.This process gives a binary output: either a shotbelongs to an anchorperson group or it does not belong toan anchorperson group. We converted this into aconfidence value in the interval between 0 and 1 asfollows.  After the candidate anchorperson clusters havebeen identified, each keyframe is compared to each of theanchorperson groups using the Cosine similarity measure,giving a visual similarity between each keyframe and eachanchorperson group.  The maximum of these values canthen be seen as a confidence that a given keyframebelongs to an anchorperson cluster, and this is output fromthe clustering algorithm for each shot.3.2. Face DetectionThe CDVP has also developed an approach forautomatically detecting faces in digital video sequences[6]. An anchorperson shot in a news programme willalways contain faces, and this information is valuable foran automatic news story segmentation system. The facedetection process is summarised in Figure 3.The first step in the face detection process involvescolour analysis of the images.  Since the colour of humanskin falls within a relatively narrow band of the colourspectrum, it is possible to detect skin-like pixels.Morphological filtering is then used to obtain smoothedhomogeneous areas of connected pixels. Shape and sizeheuristics remove some of these candidate regions.  Anycandidates remaining are rescaled to 64 x 64 pixels andpassed to a principal component analysis (PCA) module.The PCA process examines candidate regions andclassifies them as faces or non-faces.  This approach usesa number of training images to form a set of basis vectors,known as Eigenfaces, spanning an optimal subspace of theinput space such that the mean square error between theprojection of the training images onto this subspace andthe original training images is minimised. Candidate faceregions can now be analysed using these Eigenfaces viatwo distinct distance measures. Distance from Face Space??????????(DFFS) measures the distance between a test image andits reconstruction using only the first N Eigenfaces, andDistance in Face Space (DIFS) measures the distancebetween a test image and the average face.  Both of thesedistance measures can be combined to give a confidencevalue indicating whether an image is a face or not.Figure 3. The face detection process.This analysis is carried out on every 12th frame in eachshot, and the confidence value for the presence of ahuman face in a shot is the average confidence value forall frames processed within that shot.3.3. Motion Activity AnalysisSince anchorperson shots tend to have low visual activitywith the only motion usually corresponding to theanchorperson’s head/body/mouth moving, we measurevisual activity across frames to aid our anchorpersondetection. To measure visual activity in a shot, we analysethe motion vectors in the MPEG-1 bitstream.  Ourapproach is similar to that of Sun et al [7], extending theapproach to represent the motion over a shot rather thanonly in individual P-frames.  We count the number ofmotion vectors in the frame whose length is less than athreshold. Only the motion vectors for the P-macroblocksare considered, since I-blocks have zero length motionvectors but do not represent zero motion.  An activitymeasure is calculated as the ratio between the number ofthese short length P-blocks and the total number ofmacroblocks in a frame.We use two separate techniques to extend thisapproach to extract a measure to represent the activityover a shot. For the first approach, the P-frame with theleast amount of activity is used to represent the shot. Thisincreases the likelihood of a shot being represented ashaving low activity: in the context of anchorpersondetection this means that we are less likely to rule outcandidate anchor shots because noise in the data makesthem seem more active than they really are.A second approach for representing the motion withina shot is to sum the motion vectors in all P-frames in ashot to represent the overall movement.  A cumulativemotion vector for each macroblock location is calculatedas the sum of the motion vectors for all P-frames in theshot at that location, as illustrated in Figure 4. In ananchorperson shot the only motion tends to be the mouthor possibly the body movement of the anchorperson,which cancels itself as the anchorperson leans first in onedirection and then another.  This motion should be wellrepresented by cumulative motion vectors, with motionvectors pointing in opposite directions cancelling eachother out over the duration of anchor shots.  We cancalculate an activity measure for the entire shot usingthese cumulative motion vectors in the same way as for anormal P-frame.0,-1 0,1 -3,50,0 0,0 4,3-2,1 1,-1 1,00,1 1,0 -2,43,0 0,0 0,0-2,1 0,1 0,10,0 1,1 -5,93,0 0,0 4,3-4,2 1,0 1,1+ =frame a frame b cumulative frame:frame a + frame b0,-1 + 0,1= 0, 0Figure 4. Calculating Cumulative Motion Vectors.4. COMBINATION OF EXTRACTED FEATURESUSING A SUPPORT VECTOR MACHINESupport Vector Machines (SVMs) are a recentlydeveloped machine learning algorithm [8]. They haveproven very popular in pattern recognition and haveshown good performance in a variety of application areas.The approach works by finding the separating hyperplanebetween input classes that has the maximum distance fromeach class. This reduces the risk of error when classifyingpreviously unseen test examples. It has the feature that thedata is only represented as a dot product between inputpoints. This allows for the dot product to be replaced by akernel function (the dot product in a feature space) tocater for datasets with a non-linear separation.For our work in news story segmentation, we used anSVM to combine the outputs of the feature analysisdescribed in Section 3. From the TRECVID2003development set [4], a subset of 20 programmes wasselected as the training set, representing 10 hours of videofootage. Each shot within these programmes isrepresented by a vector of the extracted features. A shotwas labeled as a positive example if the shot contains astory boundary, and negative otherwise. In addition to thefeatures described, the length of each shot, measured inframes, is also input to the SVM, since anchorperson shotsintroducing a new story tend to be longer than others.The resulting SVM classifier, trained using thepublicly available SVM-light system [9], can be used todetect the anchorperson shots in a broadcast. We then logstory boundaries as the beginning of these anchorpersonshots.5. EVALUATIONWe evaluated our system against the TRECVID2003test collection, which had been manually marked up forstory boundaries, using the standard evaluation measuresof precision and recall. The TRECVID2003 collectionFiltered image after morphological adjustmentImage after applying size/shape heuristicsDetected faces with confidence scoreFor every 12    Frame th0.70.5 0.80.2??????????contains broadcast TV news from CNN and from ABC. Inorder to test the robustness of our feature extraction tovariations in the production styles of differentbroadcasters, we trained a number of systems and testedeach against the complete TRECVID2003 test set (bothCNN and ABC):1. A generic system was trained using both CNN andABC material and evaluated on a collection consistingof both CNN and ABC.2. A system was trained using only ABC data and wasevaluated on a collection of both CNN and ABC.3. A system was trained using only CNN material andwas evaluated on a collection of both CNN and ABC.4. We also tested the shot clustering algorithm run inisolation as a story bound detector. For this system anycandidate anchor shots identified by the clusteringalgorithm are logged as story bounds, withoutincluding the other extracted features.Initial results were poor (0.33 recall and 0.43precision) but by adding a random element to theclustering process we were able to achieve animprovement in precision and recall. The results for eachsystem are shown in Table 1.All combined systems outperformed the clusteringalgorithm run in isolation, giving a good increase inprecision for a relatively moderate loss in recall, clearlyshowing that combining features using a Support VectorMachine does give an improvement in performance.Surprisingly, each of the systems trained on a singlebroadcaster (systems 2 and 3) actually outperform thesystem trained on both broadcasters (system 1): this issomething we intend to investigate further at a later stage.System Recall Precision1 0.4298 0.53492 0.4647 0.51933 0.4794 0.51104 0.519 0.4030Table 1. Results for News Story Segmentationsystem against the TRECVID2003 test set.6. CONCLUSIONS AND FUTURE WORKThe results presented in the previous section are slightlydisappointing, particularly considering that otherparticipants in the TRECVID2003 Story Segmentationtask achieved 0.71 precision and 0.76 recall for runsinvolving audio visual analysis only (i.e. no text analysis).This disparity can be partly explained by the fact that oursystem uses no audio features, only video, unlike othersystems it is being compared with, and our system usedless visual features than others. Our main aim has been thedevelopment of a generic segmentation system that isrobust to variations in production styles between differentbroadcasters. The fact that we can train our system on onebroadcaster and test on another without any loss ofperformance is very encouraging in this respect. The newsstory segmentation system described in this paper hasbeen running daily as part our Físchlár-News system [1]and has recently shown its versatility when the nationalTV broadcaster changed its TV news production stylewith new studio setting, etc.Our system is clearly dependent on the results of theshot clustering algorithm and the other features essentiallyact as filters to its output. We plan to make this algorithmmore robust by using more features than just colourhistograms to represent shots in the clustering process.Also, we plan to introduce more diverse feature extractiontools, so that our system is not so dependent on onealgorithm. This includes text-based segmentation usingclosed captions, and in the audio analysis domain we willincorporate a speaker segmentation tool and a silencedetection tool.Acknowldgements. This material is based upon worksupported by the IST programme of the EU in the projectIST-2000-32795 SCHEMA.7. REFERENCES[1] Smeaton, A.F., Murphy, N., O’Connor, N., Marlow, S., Lee,H., McDonald, K., Browne, P., Ye, J. “The Físchlár DigitalVideo System: A Digital Library of Broadcast TV Programmes”,JCDL 2001, ACM + IEEE Joint Conference on DigitalLibraries, Roanoke, VA, 24-28 June 2001.[2] Zhang, H.J., Low, C., Smoliar, S., Wu J. “Video parsing,retrieval and browsing: an integrated and content-basedsolution”, Proceedings of ACM Multimedia, New York, 1995.[3] Zhang, H., Gong, Y., Smoliar, S., Tan, S. “AutomaticParsing of News Video”. Proceedings of the IEEE InternationalConference on Multimedia Computing.and Systems, 1994.[4] Guidelines for the TRECVID 2003 Evaluation. http://www-nlpir.nist.gov/projects/tv2003/ (last visited January 2004)[5] O'Connor, N., Czirjek, C., Deasy, S., Marlow, S., Murphy,N., Smeaton, A.F. “News Story Segmentation in the FíschlárVideo Indexing System”, Proceedings of the InternationalConference on Image Processing (ICIP 2001), Thessaloniki,Greece , 10-12 October 2001.[6] Czirjek, C., O'Connor, N., Marlow, S., Murphy, N.  “FaceDetection and Clustering for Video Indexing Applications”,ACIVS 2003 - Advanced Concepts for Intelligent Vision Systems,Ghent, Belgium , 2-5 September 2003.[7] Sun, X., Manjunath, B.S., Divakaran, A. “Representation ofmotion activity in hierarchical levels for video indexing andfiltering”, Proceedings of IEEE International Conference onImage Processing (ICIP), Rochester, NY, USA, September 2002.[8] Burges, C., “A Tutorial on Support Vector Machines forPattern Recognition”, Data Mining and Knowledge Discovery,2(2):121-167, 1998.[9] Joachims, T. “Making large-Scale SVM Learning Practical”,Advances in Kernel Methods - Support Vector Learning,Schölkopf , B., Burges, C., Smola, A., (eds.), MIT-Press, 1999.??????????

A Tutorial on Support Vector Machines for Pattern Recognition”, Data Mining and Knowledge Discovery,

Automatic Parsing of News Video”.

Face Detection and Clustering for Video Indexing Applications”,

Guidelines for the TRECVID

Making large-Scale SVM Learning Practical”,

News Story Segmentation in the Físchlár Video Indexing System”,

Representation of motion activity in hierarchical levels for video indexing and filtering”,

The Físchlár Digital Video System: A

Video parsing, retrieval and browsing: an integrated and content-based solution”,

A generic news story segmentation system and its evaluation

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Crossref

Irish Universities

Name not available