We present a method for merging multiple partitions into a single partition, by minimising the ratio of pairwise agreements and contradictions between the equivalence relations corresponding to the partitions. The number of equivalence classes is determined automatically. This method is advantageous when merging segmentations obtained independently. We propose using this consensus approach to merge segmentations of features tracked on video. Each segmentation is obtained by clustering on the basis of mean velocity during a particular time interva

Cohn, AG

Fraile, R

Hogg, DC

English

White Rose Research Online

promoting access to White Rose research papers    White Rose Research Online eprints@whiterose.ac.uk    Universities of Leeds, Sheffield and York http://eprints.whiterose.ac.uk/    This is an author produced version of a paper published in International Conference on Pattern Recognition  White Rose Research Online URL for this paper:  http://eprints.whiterose.ac.uk/75558/    Published paper: Fraile, R, Hogg, DC and Cohn, AG (2008) Motion segmentation by consensus. In: International Conference on Pattern Recognition (ICPR 2008), 7-11 December 2008, Tampa, Florida. IEEE Press, 241 - 244.  http://dx.doi.org/10.1109/ICPR.2008.4760999  Motion Segmentation by ConsensusRoberto Fraile, David C. Hogg and Anthony G. CohnSchool of ComputingThe University of LeedsUnited KingdomEmail: rf@comp.leeds.ac.ukAbstract—We present a method for merging multiple partitionsinto a single partition, by minimising the ratio of pairwiseagreements and contradictions between the equivalence relationscorresponding to the partitions. The number of equivalenceclasses is determined automatically. This method is advantageouswhen merging segmentations obtained independently. We proposeusing this consensus approach to merge segmentations of featurestracked on video. Each segmentation is obtained by clusteringon the basis of mean velocity during a particular time interval.I. INTRODUCTIONMotion is an important cue for object segmentation andfinding regions of similar motion is an important step forcomputer vision applications such as object detection andtracking.Pixels corresponding to objects in motion can be groupedon the basis of their appearance and relative motion [1].Appearance models of dense features have been used for two-layer segmentation [2] (where each layer is a set of featuresthat follow the same motion in the image plane) and multiple-layer segmentation [3]. Such global methods minimise anenergy functional constructed from local appearance, imposinga constraint on the motion. For example, in [4], the constraintis given by a Hidden Markov Model of actions, alternatingbetween segmentation and model fitting. Another constraintis that a higher derivative of the position of each featureis bounded, effectively requiring smooth motion. But denseoptical flow is a highly redundant representation of motion.Feature trackers [5]–[8] effectively reduce the amount ofmotion to process to a few features chosen to maximise thestability of their appearance along time. Common fate can beused to group features tracked into objects.Relative motion models have been used by [9], dependingonly on the position of tracked features, and not on theirappearance. Groups of features were split and merged usingan on-line update which assumed features from the sameobject undergo an affine transformation between frames, at thecost of sensitivity to tracking errors and density of features.Each feature group had an associated reference frame, andthe affine transform was defined with respect to that frame.Features were segmented differently during initialisation andupdate. Such approaches will fail when there is a lack ofsupport for affine grouping, for example when the geometrictransformations involved are degenerate, or the features areshort lived and there is not sufficient overlap in their supportintervals.In [10] a consensus of segmentations of an image’s pixelswas achieved by constructing a matrix that counted the numberof times two elements were grouped together, effectivelybuilding a histogram of instances of an equivalence relation.Their approach included a statistical measure of consensus, inorder to choose the number of clusters that achieved the bestconsensus. This approach for merging multiple segmentationsinto one can in principle be applied to any other kind ofsegmentation, including the segmentation of sparse features.We propose an algorithm that takes feature trajectories anduses the mean velocity over different time intervals to producea family of independent segmentations that capture motion ata variety of granularities. This is in contrast with [9], wheregroups of features were formed and destroyed on-line, in orderto allow the detection of groups of objects moving at differentvelocities. Each of our segmentations corresponds to motionsaliency at different time scales, and, because features arerelatively short lived, segmentations are defined on overlappingsets of features.The reason why time intervals of multiple lengths areconsidered is that sets of features which move together overshort timescales may separate over larger timescales due toobjects, parts of objects or groups of objects moving apart. Incontrast, objects, parts or groups which remain together overa long timescale may exhibit different short-term motion.A binary partition is then obtained for each time intervalby applying the k-means algorithm, using Euclidean distance,to an estimated mean velocity. Multiple partial segmentationsare then merged into a single segmentation by a consensusalgorithm. Each segmentation is considered as an equivalencerelation defined on a subset of features, over a particular timeinterval. Features which should fall in the same class butdo not appear in the same segmentations, are linked throughtransitivity. For example, if features a and b appear in the sameclass in frame 100, and features b and c appear in the sameclass in frame 110, then features a and c should belong to thesame class, unless there is a segmentation which separatesthem. The consensus algorithm approximates the simplestsegmentation which does not group together features whichwere separated in any of the partitions. Although each partitionhas only two classes, the consensus algorithm generates asmany classes as needed.Section II describes a method to segment features on thebasis of their mean velocity at a particular time interval.Section III presents the consensus method, which can beapplied to any kind of partial set partitions. The impliedmotion model is that an object is a maximal set of features thathave a distinct velocity at different time scales, requiring noother geometric spatio-temporal constraints such as Euclideanor affine similarity. Experiments on videos of people walkingin an underground station are used to illustrate the algorithm.II. SEGMENTATION OF A TIME INTERVALFeature tracks are grouped together on the basis of theirestimated mean velocity during a time interval. In this sectionwe explain how a segmentation is produced from a number ofsuch tracks, {pi}, given an interval (t− , t+ ).Each track consists of temporally and spatially contiguouscoordinates. In practice, this is the position at time t, p(t), fora time interval. The mean velocity vi(t, ) of each track pi inthe interval (t− , t+ ) is computed by finite differences onthe features’ positions in frames t−  and t+ .vi(t, ) ≈ pi(t+ )− pi(t− )2The velocity estimates {vi} are partitioned into two sets usingk-means on the Euclidean distance. In order to favour segmen-tations with the smallest number of groups, the parameter kin the k-means algorithm was set to 2, the number of groupswill typically be greater than 2 after consensus.Time intervals (t− , t+ ) are varied in centre and width.Different interval centres t are chosen in order to containoverlapping sets of features, and different interval radii  arechosen to measure the mean velocity of features over differentscales. The choice of intervals will therefore depend on thelifespan of the trajectories produced by the feature tracker.Each of those time intervals will yield a partial segmentationPj which is not defined on the entire set of features but ona subset Sj . Features which belong to the same object inmotion but do not appear in the same frame, should be groupedtogether by transitivity on their relation with common features.This is done by the consensus algorithm.III. CONSENSUS OF SEGMENTATIONSIn this section we describe how the family of segmentations{Pj} is merged into a single segmentation. In [10] the sumof adjacency matrices for all Pj was computed. We normalisethis sum over the number of support sets Sj in which the twofeatures in a pair are present. We will also simplify the choiceof number of segments by introducing a threshold over theresulting matrix, and then computing an approximation of itstransitivity closure [11].Each segmentation Pj corresponds to an equivalence rela-tion ∼j defined over the features Sj for which the segmenta-tion is defined. Two elements a and b in Sj are related, andwe denote it as Pj(a) = Pj(b) if they belong to the same classin Pj . The purpose is to define a single equivalence relation∼ over all the features ∪jSj , which is maximally consistent,in some sense, with the relations ∼j .A family of equivalence relations {∼j} is consistent overtwo elements a and b when the ratio of number of relations isuch that a ∼j b over the number of relations such that a 6∼j bis above a threshold λ.The criterion for relation consensus is to maximise the con-sistency of the merged relation with each of the contributingrelations, and also maximise the number of relation instancesin the merged relation (which corresponds to minimising thenumber of equivalence classes or segments).a) Consensus algorithm: Given a set Ω, subsets{Sk}nk=1, and a partition Pk of each Sk, Pk : Sk → {1, 2, . . .},produce a partition P of Ω that approximates the transitivityclosure of thresholded ratios of pair-wise agreements overcontradictions with each partition Pk:1) define the matrix A asaij =| {k : i, j ∈ Sk, Pk(i) = Pk(j)} |2) define the matrix B asbij =| {k : i, j ∈ Sk, Pk(i) 6= Pk(j)} |3) for 0 < λ ≤ 1, define element-wise Cλ = (AB ≥ λ) fora scalar threshold λ.4) Compute possibly overlapping family of sets D, formedby the equivalence classes Xk defined in the followingway: let G0 = Ω, by iterating for increasing k until Gkis empty:a) choose an element gk from Gk.b) compute Xk as the equivalence class of gk in Ω.c) define Gk+1 = Gk −Xk5) Compute P as the partition given by the the maximalnon-overlapping subsets Ek = Gk − ∪l 6=kGl.Note that this algorithm is sensitive to the choice of represen-tatives gk in step 4.a.The partitions being defined over non-overlapping sets Sk,and therefore features which are not directly related by anyparticular segmentation might end up in the same equivalenceclass, this is achieved by computing the transitivity closure ofthe final relation (Step 4) which is enforced by grouping intosets Xk. But, in order to avoid one single misclassified featureto join two equivalence classes together, features which appearin two different classes are removed (Step 5). The thresholdλ can be chosen, for example, to minimise the number offeatures dropped in this way.IV. EXPERIMENTSA sequence from the i-LIDS dataset, made available forresearch by the UK Home Office [12], were processed usingan implementation of the KLT feature tracker [5], [7], [8] withdefault settings, requesting 700 features. Affine checks wereswitched off in order to maximise the length of feature tracks.The video was filmed using a public transport surveillancecamera. It shows pedestrians walking on an undergroundplatform and trains arriving and departing. Which regionswere segmented out depended on the radius  of the samplinginterval (t − , t + ) around frame t, as shown on Figure 1.Applying the consensus algorithm to segmentations obtainedwith fixed t and increasing , produces a segmentation whichFig. 1. Segmentation of the mean velocity computed over the interval centredon frame 50 with  = 2 .Fig. 2. Number of features dropped for each choice of λ(scaled× 100).separates different objects in motion. Trains were segmentedout for low values of , typically between 2 and 6 frames.Pedestrians were segmented out for higher values of , typ-ically around 12 frames. The number of features segmenteddecreases exponentially for increasing , and features on faster-moving objects exist during a narrower interval. Pedestriansare segmented out for values of  for which there are no trainfeatures.The number of features for a particular segmentation canbe increased by applying the consensus algorithm for variousvalues of t. For a fixed , consensus of segmentations withvarying t produces binary segmentations showing a particularobjects in motion: a train or a group of pedestrians. Then,consensus is applied again to produce a segmentation intomultiple classes that separates the multiple objects in motion.A rational choice of threshold λ in the consensus algorithmwould be a value that minimises the number of featuresthat are discarded. In the experiments it was observed thatsuch a choice corresponded to good segmentations. Figure 2shows the number of features dropped as a function of λwhen performing consensus of the 11 partitions with  = 24,t = 1000 and t offset by −5,−4, . . . , 4, 5. The value λ = 0.5was used in the consensus algorithm when computing thebottom-right segmentation of Figure 3.V. CONCLUSIONWe have presented an algorithm for segmentation of trackedfeatures which does not rely on rigidity assumptions and usesthe output of an off-the-shelf feature tracker and for whichthe features in each segmented region have similar meanvelocities over different scales. It first produces a family ofpartial partitions of the sets of features, each of them definedover a time interval. It then selects automatically the numberof objects in motion. It uses an algorithm for consensus ofmultiple segmentations, which effectively obtains new relationinstances between unrelated features by applying transitivity.The underlying motion model only requires that featuresbelonging to the same object have a distinct mean velocityat different time intervals, avoiding rigidity constraints. Thisconsensus method is advantageous when exploiting the infor-mation from data partitions obtained independently and thenumber of clusters is unknown.Future work will be directed towards making a well-foundedchoice of sampling intervals.REFERENCES[1] R. Tron and R. Vidal, “A benchmark for the comparison of 3-D motionsegmentation algorithms,” in CVPR, 2007.[2] A. Criminisi, G. Cross, A. Blake, and V. Kolmogorov, “Bilayersegmentation of live video,” in IEEE Computer Vision and PatternRecognition or CVPR, 2006, pp. I: 53–60. [Online]. Available:http://dx.doi.org/10.1109/CVPR.2006.69[3] N. Jojic and B. J. Frey, “Learning flexible sprites in videolayers,” in CVPR. IEEE Computer Society, 2001, pp. 199–206.[Online]. Available: http://doi.ieeecomputersociety.org/10.1109/CVPR.2001.990476[4] L. Gui, J.-P. Thiran, and N. Paragios, “Joint object segmentationand behavior classification in image sequences,” in CVPR. IEEEComputer Society, 2007. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2007.383234[5] J. Shi and C. Tomasi, “Good features to track,” in IEEE Conferenceon Computer Vision and Pattern Recognition (CVPR’94), Seattle, Jun.1994. [Online]. Available: citeseer.nj.nec.com/shi94good.html[6] M. Grabner, H. Grabner, and H. Bischof, “Learning features fortracking,” in IEEE Computer Vision and Pattern Recognition or CVPR,2007, pp. 1–8. [Online]. Available: http://dx.doi.org/10.1109/CVPR.2007.382995[7] J.-Y. Bouguet, “Pyramidal implementation of the lucas kanade featuretracker: Description of the algorithm,” Jean-YvesBouguet, 2002.[8] S. Birchfield, “KLT: An implementation of the Kanade-Lucas-Tomasifeature tracker,” http://www.ces.clemson.edu/∼stb/klt/, 2007.[9] S. Pundlik and S. Birchfield, “Motion segmentation at any speed,”in Proceedings of BMVC, vol. I, 2006, p. 427. [Online]. Available:http://www.bmva.ac.uk/bmvc/2006/papers/226.pdf[10] S. Monti, P. Tamayo, J. Mesirov, and T. Golub, “Consensus clustering. Aresampling based method for class discovery and visualization of geneexpression microarray data,” Machine Learning, vol. 52, no. 1-2, pp.91–118, 2003.[11] S. S. Skiena, The Algorithm Design Manual. pub-SV:adr: Springer-Verlag, 1998.[12] Home Office Scientific Development Branch, “Imagery library forintelligent detection systems (i-LIDS),” http://homeoffice.gov.uk, 2007.Fig. 3. Segmentation (top and middle) and consensus (bottom) for frames 1000 (left) and 1250 (right).

Motion segmentation by consensus

Abstract

Similar works

Full text

Available Versions

White Rose Research Online

Crossref