This paper deals with the representation of video sequences useful
for tasks such as long-term analysis, indexing or browsing. A Table
Of Content and index creation algorithm is presented, as well as
additional tools involved in their creation. The proposed method
does not assume any a priori knowledge about the content or the
structure of the video. It is therefore a generic technique. Some
examples are presented in order to assess the performance of the
algorithmPeer ReviewedPostprint (published version

Llach, J

Salembier Clairon, Philippe Jean

English

UPCommons. Portal del coneixement obert de la UPC

ANALYSIS OF VIDEO SEQUENCES: TABLE OF CONTENTS AND INDEX CREATIONJoan Llach, Philippe SalembierUniversitat Polite`cnica de Catalunya, Campus Nord - Mo`dul D5, C/ Gran Capita` s/n, 08034 Barcelona, SpainE-mail: fjllach,philippeg@gps.tsc.upc.es — WWW: http://gps-tsc.upc.es/imatgeABSTRACTThis paper deals with the representation of video sequences usefulfor tasks such as long-term analysis, indexing or browsing. A TableOf Content and index creation algorithm is presented, as well asadditional tools involved in their creation. The proposed methoddoes not assume any a priori knowledge about the content or thestructure of the video. It is therefore a generic technique. Someexamples are presented in order to assess the performance of thealgorithm.1. INTRODUCTIONIn the framework of video analysis for indexing application (for ex-ample application related to MPEG-7), the representation of videosequences is an important issue. It is not enough to describe thecontent of a video but also to develop techniques which are ableto automatically create these descriptions. In this paper, a new tech-nique that automatically creates TOCs and indexes is presented. Theproposed algorithm only relies on visual information, unlike othertechniques which use both video and audio information [3].The goal of the TOC is to define the structure of the video se-quence in a hierarchical fashion. The original sequence should besubdivided in sub-sequences which can also be divided in shortersub-sequences, etc. At the end of this division process, the short-est entity to be described is the micro-segment. The index is alsoa hierarchical structure but it does not describe the structure of thesequence but the occurrences of similar content.The creation of the TOC and index follows the same strategy.The first step splits the sequence into shots. The second step divideseach shot into micro-segments which are sub-components of a shotwhere the camera activity is homogeneous. These micro-segmentsconstitute the lowest level of the TOC or index representations. Thethird step creates the hierarchical structures by clustering the de-tected shots.This paper is structured as follows. Section 2 deals with the shotdetection algorithm. The temporal segmentation of shots is detailedin section 3, whereas their clustering is explained in section 4. Someconclusions are driven in section 5.2. SHOT DETECTIONThe first step in the TOC and index creation splits the sequenceinto shots. Taking into account that a shot is a set of contiguousframes without editing effects, the algorithm has to detect the tran-sitions between consecutive shots. These transitions can be abruptor more sophisticated, like dissolves or fades. The shot detectionalgorithm consists of two main steps: computation of the mean Dis-placed Frame Difference mDFD curve and its segmentation intoshots.This work is supported by Philips Research.5010015020025030035021000 21100 21200 21300 21400 21500 21600 21700 21800 21900 22000mDFDFilteredMarkersDissolveScene cutScene cutScene cutScene cutS1S2S3 S4 S5 S6 S7 S8 S10S9Figure 2: mDFD for the news2 sequence (frame 21 000 to 22 000,thin line). The filtered curve (thick line) and the detected markers(horizontal segments) are also plotted. S1–S10 depict the limits ofthe detected shots.2.1. Computation of the mDFD curveThe mDFD curve is obtained using a variation of the DisplacedFrame Difference DFD to take into account luminance and chromi-nance information. If Ixand Iyare the image dimensions and w’sare the weights for Y; U; V components, the mDFD is computed asfollows: mDFD(t) = 1IxIyPY;U;VkwkPIx;Iyi;jjDFDk(i; j; t 1; t+ 1)j.In the example shown in Fig. 2 the weights have been set tofwY; wU; wVg = f1; 3; 3g, which are typical values. The high-est peaks of the curve correspond to the abrupt transitions (frames21 100, 21 195, 21 633 and 21 724). On the other side, the oscillationfrom frame 21 260 to 21 279 corresponds to a dissolve. The presenceof large moving foreground objects in frames 21 100–21 195 and21 633–21 724 creates high level oscillations of the mDFD curve.2.2. Segmentation of the mDFD curveThe second block of the shot detection algorithm detects the videoediting effects. Our approach has been to adapt classical spatial seg-mentation strategies to the case of temporal segmentation. Most ofclassical shot detection techniques use threshold-based segmenta-tion to extract the highest peaks of the mDFD or another type ofmono-dimensional curve. Although a large number of shots can ac-tually be detected by this approach, this class of algorithms is not ro-bust and quite sensitive to noise. It is in particular difficult to detectpeaks of small contrast corresponding to fading or special effects.The solution that we have used is a homogeneity-based approachrelying on morphological tools. The mDFD curve is processed fol-lowing four steps:0–306 307–362 363–663 664–706 707–801 802–1091 1092–1145 1146–12791280–1376 1377–1476 1477–1508 1509–1663 1664–1834 1835–1863 1864–1945 1946–2020Figure 1: Detected shots for drama sequence, using lmin= 20 and c = 10. Each image represents the central frame of the shot, the numbersshow its starting and ending frame. Shots are ordered from left to right and from top to bottom.1. Simplification by temporal filtering. Morphological closingwith a 1D structuring element of length lmin. With this op-eration, negative peaks the length of which is less than lminframes are removed. The lminparameter defines the durationof the shortest shot to be detected.2. Simplification by positive contrast filter. Peaks that have a pos-itive contrast lower than c parameter are removed.3. Marker extraction. Each marker corresponds to the kernel ofone shot. So, each marker must cover a portion of the curvewith a high probability to belong to a single shot. Becausecontiguous frames that belong to the same shot are quite simi-lar, the value of the mDFD will be small around those frames.Thus, to extract the markers a negative contrast filter is usedbecause it detects the relative minimums of the curve.4. Watershed. Its purpose is to propagate the markers on themDFD curve until all its points are assigned to a marker. Thepropagation process is performed by applying a watershed al-gorithm on the curve using as initial markers those obtained inthe previous step.Figure 2 shows the filtered curve (both temporal filter and positiveand negative contrast filter), the resulting markers and detected shotsusing lmin= 10 and c = 10. Even though some over-segmentationappears around frames 21 150 and 21 700, both the scene cuts andthe dissolve have been correctly detected. The over-segmentation isnot a problem because the next steps of the algorithm eliminate thiseffect.3. TEMPORAL SEGMENTATION OF VIDEO SHOTSUSING CAMERA MOTION PARAMETERSThe goal of the temporal segmentation algorithm is to split eachshot into several micro-segments which present a high level of ho-mogeneity on the camera motion parameters. The data used to per-form the segmentation are the camera motion parameters currentlyused in MPEG-7 [1]. The algorithm is applied to each shot andconsists of two steps: 1) each shot is over-segmented into severalmicro-segments which must present a perfect homogeneity (see sec-tion 3.1), and 2) a merging process is applied while the homogeneitylevel of the set of micro-segments is below a predefined threshold.In order to segment the shot, it is necessary to define a distanceto compare micro-segments and a parameter which allows the as-sessment of the quality of a micro-segment or a partition (i.e., set ofmicro-segments). In both cases, we use a motion histogram, definedas follows:Hs[i] =NiLs; i 2 fPanLeft, ZoomIn, Fix, . . .g (1)where s represents the label of the segment inside the shot, i the mo-tion type, Lsthe length of segment s and Nithe number of framesof segment s with motion type i. Note that each bin of the his-togram H shows the percentage of frames with a specific type ofmotion.PHs[i] may be higher than 1:0, because different motionscan appear concurrently.3.1. HomogeneityWe assume that a segment is perfectly homogeneous when itpresents a single combination of camera motion parameters alongall its frames. A segment is not homogeneous when it presents im-portant variations on these parameters. The segment homogeneityis computed on its histogram (Eq. 1). If a segment is perfectly ho-mogeneous, the histogram bins are equal to either 1.0 or 0.0. If asegment is not perfectly homogeneous, the bins can present inter-mediate values.To measure the segment homogeneity, we measure how much itshistogram differs from the ideal one. The distance corresponding tobins with high values is the difference between the bin value and 1.0.Analogously, for bins with small values, the distance is the bin valueitself. Mathematically, The homogeneity of a segment s is given by:H(s) =Xi(i); where (i) =1:0  Hs[i]; if Hs[i]  0:5Hs[i]; if Hs[i] < 0:5(2)where Hsis the histogram of the segment s and i indicates the dif-ferent motion types. The homogeneity of a shot S is equal to the ho-mogeneity of its segments weighting by the length of each of them.Note that small values of H correspond to high levels of homogene-ity.The distance between two segments (s1; s2) is the homogeneityof the union of the segments d(s1; s2) = H(s1[ s2).3.2. Temporal segmentation algorithmThis algorithm consists of the following steps:1. Initial over-segmentation. In this step, the shot is over-segmented in order to obtain a set of perfectly homogeneousmicro-segments. Mathematically, the following conditionmust be fulfilled: H(s) = 0; 8s 2 S.Low detailHigh detailFigure 3: Example of temporal segmentation of shot #8 of MPEG-7nhkvideo sequence. Above: camera motion parameters chronogram;below: two segmentations.2. Fusion order. In this step, the distance between all neighbor-ing segments (temporally connected) is computed. The closestpair of segments is then selected for possible merging in thenext step.3. Fusion criterion. To decide if the selected pair of segmentsare going to be merged, we compute the homogeneity of theshot assuming than the minimum distance segments have al-ready been merged and then the following criterion is applied:merge, if H(S)  H; do not merge, if H(S) > H.Note that the fusion criterion is global: the decision depends onthe homogeneity of the resulting partition and not —exclusively—on the homogeneity of the resulting segment. If the merging is done,a new iteration starts at the second step of the algorithm. The merg-ing process ends when there is no pair of neighboring segments thatcan be merged. The algorithm allows different levels of detail (seeFig. 3).4. SHOT CLUSTERINGThe shot clustering process is divided into two parts: shot mergingand tree structuring. In the first step, pairs of shots are groupedtogether creating a binary tree. In the second step, the binary treeis restructured in order to reflect the similarity present in the videosequence.4.1. Shot mergingThe shot merging algorithm yields a binary tree which representsthe merging order of the initial shots (see section 2). In this tree, theleaves represent the initial shots, the top node represents the wholesequence and the intermediate nodes represent sequences that arecreated by the merging of several shots. The merging criterion isdefined by a distance between shots, merging first the closest shots.In order to compute the distance between shots it is necessary todefine a shot model that provides the features to be compared, andto set the neighborhood links between them, which indicate whatmerging can be done. The TOC and index creation only differ inthe neighborhood links. The former sets a link between each pair oftemporally connected shots, the latter between all pair of shots.The process ends when all the initial shots have been merged intoa single node or when the minimum distance between all couplesof linked nodes is greater than the specified threshold [5]. In thelatter case, not a single tree but a set of binary trees —a forest— isobtained.4.1.1. Shot and sequence modelThe shot model must allow us to compare the content of severalshots to decide what shots must be merged and which is their merg-ing order. In still images, luminance and chrominance are the mainfeatures of the image [2]. In a video sequence, due to the temporalevolution, motion is an important source of information [2, 4]. So,average images, histograms of luminance/chrominance information(Y UV components) and motion information (x and y componentsof motion vectors) are used to model the shots.4.1.2. Merging distanceIn order to compute the distance between a pair of nodes, three dif-ferent cases must be considered depending on the type of nodes. Atthe beginning of the merging process, the nodes represent a singleshot: the average of the Euclidean distance between the differentcomponents of the model is used. In the following steps of the pro-cess, the distance must be computed between nodes that may repre-sent more than a single shot: the minimum and maximum distancebetween all pair of shots (one from each node) is used.4.1.3. Merging algorithmThe merging algorithm consists of three preliminary steps and themerging process itself. In the first step, nodes are modeled accordingto the model previously described. In the second step, neighborhoodlinks between shots are set and in the third step the distance betweenlinked shots is computed.Since the whole algorithm must be able to create both the TOCand the index of a video sequence, the criterion used to set the neigh-borhood links differs depending on the main objective. In the TOCcreation process, neighborhood links are set according to temporalconnectivity. On the other hand, all pair of shots will be linked andtherefore may be merged in the index creation process.Once the algorithm has been initialized, the merging processstarts performing the following steps:1. Get minimum distance link. Both the minimum and the maxi-mum distance is computed for every pair of linked nodes. Be-fore to check the minimum distance, the maximum distance ischecked. If maximum distance is higher than the maximumdistance threshold dmax, then the link is discarded. Other-wise, the link is taken into account. Once all links have beenscanned, the minimum distance link is obtained.2. Check distance criterion. In order to decide if the nodespointed by the minimum distance link must be merged, theminimum distance is compared to the minimum distancethreshold dmin. If the minimum distance is higher than thethreshold, the process ends. Otherwise, pointed nodes aremerged and the process goes on.3. Update links. Links are updated to take into account the merg-ing that has been done.4. Update distances. The distance of those links which point tothe new node is recomputed.5. Check top node. If all initial shots have been merged into asingle node, the process ends. Otherwise, a new iteration be-gins.The merging process may yield a single tree or a forest dependingon the degree of similarity of the initial shots. An example of binarytree for TOC creation is shown in Fig. 4. Inside the leaf nodes ofthis tree, we have indicated its label, and between brackets [:::] thestarting and ending frame number of the shot. Inside the remain-ing nodes, we have indicated its label, between parenthesis (:::) thefusion order and between brackets the minimum and maximum dis-tance between its two siblings.4.2. Tree structuringThe purpose of this algorithm is to restructure the binary tree into anarbitrary tree that should reflect more clearly the video structure. Tothis end, nodes that have been created by the binary merging processbut that do not convey any relevant information should be removed.The criterion used to decide if a node must appear in the final treeis based on the variation of the similarity degree (distance) betweenthe shots included in the node: If the analyzed node is the root node (or one of the root nodes,if various binary trees have been obtained after the mergingalgorithm), then the node should be preserved in the final tree. If the analyzed node is a leaf node (i.e., it corresponds to ainitial shot), then it also remains in the final tree. Otherwise, the node will be kept in the final tree ifjdmin(analyzed node)   dmin(parent node)j <  andjdmax(analyzed node)  dmax(parent node)j < .As shown in Fig. 5, the tree resulting from the restructuring steprepresents more clearly the structure of the video sequence. Nodesin the first level of the hierarchy (28,12,13,21) represents the fourscenes of the sequence, while nodes in the third —or occasionallyfourth— represent the initial shots.5. CONCLUSIONSIn this paper, techniques for TOC and index creation and video shotdetection and micro-segment segmentation have been presented.The shot detection algorithm is able to cope with abrupt transitionsas well as smooth ones even though it over-segments those portionsof the video sequence with a high level of motion. In order to solvethis drawback, a confidence mask will be added to the motion vec-tors to discard those parts of the image which can not be properlyestimated when computing the mDFD.The ability of the micro-segment segmentation algorithm to com-pute partitions at different level of detail allows to adjust the gran-ularity of the resulting TOC or index to take into account differentapplications. Moreover, the algorithm is quite robust in front of mis-takes in the camera parameters.Finally, the TOC and index creation algorithm provides an uni-fied method to compute both structures. Moreover, new pieces ofinformation can easily be introduced in the merging process. Forexample, although the algorithm has been developed to deal withgeneric sequences, in the case of knowing before hand some char-acteristics of the sequence, this a priori information can be takeninto account to improve the performance of the algorithm in specificscenarios.6. REFERENCES[1] R. Jasinschi, A. She, T. Naveen, A. Tabatabai, S. Jeannin andB. Mory, “Motion Descriptors for Content Based Video Repre-sentation”, to be published in Image Communications: SpecialIssue on MPEG-7 proposals, 1999.31 (15)[2.24−3.66]30 (14)[2.28−3.85]29 (13)[2.68−3.61]28 (12)[1.49−2.85]1[0−306]27 (11)[1.49−2.44]26 (10)[1.38−2.23]2[307−362]25 (9)[1.51−2.88]24 (8)[0.73−1.93]23 (7)[0.62−1.85]22 (6)[1.71−1.71]3[363−663]4[664−706]18 (2)[0.64−0.64]5[707−801]6[802−1091]7[1092−1145]20 (4)[1.53−2.10]19 (3)[1.47−1.47]8[1146−1279]9[1280−1376]10[1377−1476]11[1477−1508]12[1509−1663]13[1664−1834]21 (5)[1.67−1.86]17 (1)[0.47−0.47]14[1835−1863]15[1864−1945]16[1946−2020]Figure 4: Binary tree created by the shot merging algorithm. Theinitial shots are shown in Fig. 1. The top node represents the wholesequence; the leaf nodes represent the initial shots.31 (15)[2.24−3.66]28 (12)[1.49−2.85]1[0−306]2[307−362]22 (6)[1.71−1.71]3[363−663]4[664−706]18 (2)[0.64−0.64]5[707−801]6[802−1091]7[1092−1145]8[1146−1279]9[1280−1376]10[1377−1476]11[1477−1508]12[1509−1663]13[1664−1834]21 (5)[1.67−1.86]17 (1)[0.47−0.47]14[1835−1863]15[1864−1945]16[1946−2020]Figure 5: Tree yielded by the tree structuring algorithm.[2] R. Mohan, “Video Sequence Matching”, in ICASSP’98, vol. 6,pp. 3697–3700, June 1998.[3] C. Saraceno and R. Leonardi, “Indexing Audio-VideoDatabases through a joint audio and video processing”, to bepublished in International Journal of Imaging Systems andTechnology.[4] H. S. Sawhney and S. Ayer, “Compact Representations ofVideos Through Dominant and Multiple Motion Estimation”,in PAMI, vol. 18, no. 8, pp. 814–830, 1996.[5] M. M. Yeung and B.-L. Yeo, “Time-constrained Clusteringfor Segmentation of Video into Story Units”, in ICPR’96,pp. 375–380, 1996.

Analysis of video sequences: table of content and index creation

This paper deals with the representation of video sequences useful

for tasks such as long-term analysis, indexing or browsing. A Table

Of Content and index creation algorithm is presented, as well as

additional tools involved in their creation. The proposed method

does not assume any a priori knowledge about the content or the

structure of the video. It is therefore a generic technique. Some

examples are presented in order to assess the performance of the

algorithmPeer Reviewe

UPCommons

https://upcommons.upc.edu/bitstream/2117/103788/1/10.1.1.63.9869.pdf

Analysis of video sequences: table of content and index creation

Abstract

Similar works

Full text

Available Versions

UPCommons. Portal del coneixement obert de la UPC

UPCommons