This paper presents the results of the crowd image analysis

challenge of the Winter PETS 2009 workshop. The evaluation

is carried out using a selection of the metrics developed

in the Video Analysis and Content Extraction (VACE)

program and the CLassification of Events, Activities, and

Relationships (CLEAR) consortium [13]. The evaluation

highlights the detection and tracking performance of the authors’systems in areas such as precision, accuracy and robustness. The performance is also compared to the PETS

2009 submitted results

Ellis, Anna

Ferryman, James Michael

Shahrokni, Ali

English

Central Archive at the University of Reading

PETS2009 and Winter-PETS 2009 Results: A Combined EvaluationA.Ellis, A.Shahrokni and J.M.FerrymanComputational Vision GroupSchool of Systems EngineeringUniversity of ReadingWhiteknights, Reading, RG6 6AY, UK{a.l.ellis| a.shahrokni| j.m.ferryman}@reading.ac.ukAbstractThis paper presents the results of the crowd image analysischallenge of the Winter PETS 2009 workshop. The evalu-ation is carried out using a selection of the metrics devel-oped in the Video Analysis and Content Extraction (VACE)program and the CLassification of Events, Activities, andRelationships (CLEAR) consortium [13]. The evaluationhighlights the detection and tracking performance of the au-thors’ systems in areas such as precision, accuracy and ro-bustness. The performance is also compared to the PETS2009 submitted results.1 IntroductionThis paper discusses the objective evaluation of the submit-ted results by contributing authors of the two PETS 2009and Winter PETS 2009 workshops on the challenges de-fined on the PETS2009 crowd dataset [12] .The theme of the Winter PETS 2009 is multi-sensorevent recognition in crowded public areas. As part of thisworkshop a challenge was set to evaluate an approach to oneor more of people counting and density estimation, track-ing, and flow estimation and event recognition, and to re-port results based on annotated datasets made available onthe workshop website [1]. In this paper the focus is track-ing and people counting challenges due to the fact that themajority of the submitted evaluations and papers were ded-icated to these tasks.In the remainder of this paper, the dataset and the groundtruth annotation details are presented in Section 2. A briefdescription of the evaluation methodology follows in Sec-tion 3, and analytic discussion of the overall performancesis provided in Section 4. Concluding remarks are given inSection 5.2 Datasets and Ground Truth2.1 DatasetsThree datasets were recorded for the workshop atWhiteknights Campus, University of Reading, UK. Fur-ther details of these datasets may be found in Ferrymanand Shahrokni [12]. The datasets comprise multi-sensor se-quences containing crowd scenarios with increasing scenecomplexity. Dataset S1 concerns person count and densityestimation. Dataset S2 addresses people tracking. DatasetS3 involves flow analysis and event recognition. In this pa-per the first two datasets are the focus.2.2 Ground TruthThe ground truth was obtained for a subsampled set offrames for each sequence with the average sampling fre-quency being 1 frame in every 3 frames.The ground truth for people counting was generated bymanually counting the people in the specified regions andthose that cross the entry and exit lines at each sampledframe.The ground truth annotation simultaneously definesbounding boxes in all views corresponding to a person, bylocating its 3D position on a discrete grid. The grid isdefined as cells of 72 in width and 132 in height on theground plane, which corresponds to an area of 24 x 44 me-tres. Errors in calibration due to the approximation of theground surface as a plane, in addition to radial distortion,and the spatial resolution of the annotation grid defined onthe ground plane, are an intrinsic part of this annotation. Inthe context of the current evaluations, further measures havebeen considered to take this into account in the evaluationprocess as described in Section 3.1978-1-4244-5504-1/09/$25.00 ©2009 IEEE3 Evaluation MethodologyThe evaluation was based on the framework by Kasturi etal. [13], which is a well established protocol for perfor-mance evaluation of object detection and tracking in videosequences. These metrics are formally used the Video Anal-ysis and Content Extraction (VACE) program and the CLas-sification of Events, Activities, and Relationships (CLEAR)consortium. As part of the PETS 2009 workshop, authorsof the representative algorithms, submitted their results inXML format using the PETS 2009 published XML Schemaavailable at [1]. These results were evaluated using the fol-lowing metrics:Notation• Gti denotes ith ground-truth object in frame t; Gi de-notes the ith ground-truth object at the sequence level;Nframes is the number of frames in the sequence• Dti denotes the ith detected object in frame t; Di de-notes the ith detected object at the sequence level• N tG and N tD denote the number of ground-truth ob-jects and the number of detected objects in frame t, re-spectively; NG and ND denote the number of uniqueground-truth objects and the number of unique de-tected objects in the given sequence, respectively• N iframes refers to the number of frames where eitherground-truth object (Gi ) or the detected object (Di )existed in the sequence• Nmapped refers to sequence level detected object andground truth pairs, N tmapped refers to frame t mappedground truth and detected object pairs• mt represents the missed detection count, (fpt ) is thefalse positive count, cm and cf represent respectivelythe cost functions for missed detects and false posi-tives, and cs = log10ID − SWITCHESt3.1 Sequence Frame Detection Accuracy(SFDA)SFDA uses the number of objects detected, the number ofmissed detections, the number of falsely identified objects,and the calculation of the spatial alignment between the al-gorithm’s output for detected objects and that of the groundtruthed objects. It is derived from a Frame Detection Accu-racy (FDA) measure. The FDA is calculated using a ratioof the spatial intersection and union of an output object andmapped ground truthed objectsOverLapRatio =Ntmapped∑i=1|Gti ∩Dti ||Gti ∪Dti |(1)FDA(t) =OverlapRatio[NtG+NtD2] (2)SFDA =∑Nframest=1 FDA(t)∑Nframest=1 ∃ (N tG ∨N tD)(3)For this study although the annotation of the groundtruth was challenging, as described in Section 2, an over-lap threshold of 100 percent for the intersection over unionscores, was used.3.2 Average Tracking Accuracy (ATA)ATA is obtained from the Sequence Track Detection Accu-racy (STDA). The STDA is a measure of the tracking per-formance over all of the objects in the sequence and fromthis ATA is defined as the sequence track detection accuracyper object. The mapping between ground truth objects anddetected objects is performed so as to maximise the mea-sure score. This metric is implemented with a hash functiondue to the fact that the track correspondence matrix to bemapped is reasonably sparse.STDA =Nmapped∑i=1∑Nframesi=1[ |Gti∩Dti ||Gti∪Dti |]N(Gi∪Di =0)(4)ATA =STDA[NG+ND2] (5)For both detection and tracking metrics in the followingdescriptions the accuracy metrics provide a measure of thecorrectness of the detections or tracks. The precision met-rics provide the measure of, in the instance where there hasbeen a correct detection or track, how close to the groundtruth that detection or track may be.3.3 Multiple Object Detection Accuracy(MODA)MODA is an accuracy measure that uses the number ofmissed detections and the number of falsely identified ob-jects. Cost functions to allow weighting to either of theseerrors are included, however for the sake of both PETS 2009evaluations they were equally set to 1.MODA = 1− cm(mt) + cf (fpt)N tG(6)3.4 Multiple Object Detection Precision(MODP)MODP gives the precision of the detection in a given frame.Again, with this metric, an overlap ratio is calculated as pre-viously defined in (1), and, in addition to a count of thenumber of mapped objects, the MODP is defined as:MODP (t) =OverLapRatioN tmapped(7)3.5 Multiple Object Tracking Accuracy(MOTA)MOTA uses the number of missed detections, the falselyidentified objects, and the switches in an algorithm’s outputtrack for a given ground truth track. These switches are cal-culated from the number of identity mismatches in a frame,from the mapped objects in its preceding frame.MOTA = 1−∑Nframest=1 (cm(mt) + cf (fpt) + cs)∑Nframest=1 NtG(8)3.6 Multiple Object Tracking Precision(MOTP)MOTP is calculated from the spatio-temporal overlap be-tween the ground truthed tracks and the algorithm’s outputtracks.MOTP =∑Nmappedi=1∑Ntframest=1[ |Gti∩Dti||Gti∪Dti|]∑Nframest=1 Ntmapped(9)In addition to the evaluation of tracking, a simple com-parison of the people count per region, against a groundtruth count per region for the sampled frames, produced theaverage percentage error in counting per region, for eachsequence.4 Evaluation ResultsAn analysis of the overall performance, of the submitted re-sults from the benchmark datasets, using the described met-rics, is described in this section. The submitted results arediverse in terms of the sequences and views used and there-fore it is not possible to draw general comparisons and con-clusions about their performance. Nevertheless, the evalua-tions presented in this section can lead to a helpful insightabout the effectiveness of different methodologies. Bothpeople counting and tracking challenges are considered.4.1 People CountingFigure 1 provides the evaluation of the counting people perregion task. Note that the y axis on this graph representsthe average error in number of people per frame, wherethe lower the value, the better the performance per frame.Table 1 gives the corresponding publication reference, foreach label, for Figure 1.Table 1: Labels and publication references for Figure 1Label ReferenceChan [10]Sharma [15]Albiol [3]Choudri [11]Alahi [2]A wide variety of methods have been proposed andtested in this category and from Figure 1 it can been seenthat the majority of the methods and their variants have con-sistent and comparable performance. While the algorithmproposed by Alboil et al. [3] remains a top performer, sev-eral methods such as Alahi et al. [2], Chan et al. [10] andChoudri et al. [11] also perform well on the more challeng-ing sequence 14-17. Further details of the variant of eachmethod can be found in their companion workshop paper.4.2 TrackingThe most tested dataset of the two PETS workshops in 2009remains S2.L1, at time sequence 12.34, for the first cameraview. Figure 3 shows how the individual algorithms per-form according to various VACE and CLEAR metrics on asingle representative camera view. Table 2 gives the corre-sponding publication reference, for each label, for Figures3, 4 and 5.Note that in the case of these metrics, higher values in-dicate better performance. It is clear that for this sequence,using MODA and MOTA as a measure, Yang et al.’s [16],Breitenstein et al.’s [9] and the linear programming-basedmethod proposed by Berclaz et al. [6] perform strongly atmultiple object detection and tracking. For precision inthis task, using MODP and MOTP, the systems describedby Berclaz et al. [7], as well the dynamic programming-based method proposed in [6] perform the strongest. Mea-suring object detection accuracy by frame and sequence us-ing SODA and SFDA metrics, the systems by Berclaz etal. [7, 6] outperform others.Figure 4 shows the median of each metric value, forall the computed views, excluding View 2 which was notprovided in the dataset, from each author. Again, the per-formance measures highlight the linear programming-basedFigure 1: Counting People in Regions.tracking algorithm [6] for multiple object detection and ac-curacy. From this figure it can be seen that although thereare variations per metric per author, the results for MOTP,MODP, SFDA, and SODA indicate a general consensus ofaccuracy.To estimate the consistency of the metrics themselvesanother evaluation is illustrated here. Figure 2 shows, foreach view, the median value of each metric for all authors.It highlights the relationship between five (SODA, SFDA,MODP, MODA, MOTP) of the six metrics used. In addi-tion it indicates the difficulty in detection and tracking fromcamera View 7. Camera View 2 was also particularly chal-lenging as it was not provided to the authors and the resultshave be obtained by re-projection from other views or the3D information. Overall this Figure shows that the metricsare consistent in their evaluation of performance.As the final evaluation, a view for each metric which cor-responds to the median value of the metric for all authors.The results are shown in Figure 5.From this Figure a fair overall performance comparisonof each algorithm and it variant forms can be inferred. Dueto its robustness to outliers, this visualisation gives a clearindication of how different algorithms perform relative toeach other. It must be noted that in some cases the accuracymeasure can be negative if the number of false positives arehigh as been seen for MOTA8 in Figure 5.5 ConclusionIt is essential that authors are able to objectively evalu-ate their detection and tracking algorithms with standard-ised metrics. The ability to compare results, with others,whether anonymous or not, provides a realistic and encour-aging research technique towards advanced, robust, real-time visual systems. In order for authors to achieve this,during the many iterative cycles of algorithm development,an online tool that they may use at their convenience seemssensible. However, in order for such a tool to be completelyautomated, the complexity of any appropriate XML filesFigure 2: Median metric values among all authors per viewmust be an absolute minimum.In addition, the use of these metrics and this study pro-vides a mechanism to highlight the strengths of the indi-vidual systems, such as accuracy, precision and robustness.It may be used for future decisions for systems placement.For example, those that require a high degree of precisionmay benefit from techniques described by authors whosesystems performed well using precision metrics.References[1] PETS: Performance Evaluationof Tracking and Surveillance.http://www.cvg.rdg.ac.uk/slides/pets.html.[2] A. Alahi, L. Jacques, Y. Boursier, and P. Van-dergheynst. Sparity-driven people localization algo-rithm: Evaluation in crowded scenes environments. InTwelfth IEEE International Workshop on PerformanceEvaluation of Tracking and Surveillance, 2009.[3] A. Albiol, M. J. Silla, A. Albiol, and J. M. Mossi.Video analysis using corners motion analysis. InEleventh IEEE International Workshop on Perfor-mance Evaluation of Tracking and Surveillance, pages31–37, 2009.[4] D. Arsic, A. Lyutskanov, G. Rigoll, and B. Kwolek.Multi camera person tracking applying a graph-cutsbased foreground segmentation in a homographyframework. In Twelfth IEEE International Workshopon Performance Evaluation of Tracking and Surveil-lance, 2009.[5] L. Bazzani, D. Bloisi, and V. Murino. A comparisonof multi hypothesis kalman filter and particle filter formulti-target tracking. In Eleventh IEEE InternationalWorkshop on Performance Evaluation of Tracking andSurveillance, 2009.[6] J. Berclaz, F. Fleuret, and P. Fua. Multiple objecttracking using flow linear programming. In TwelfthFigure 3: Performance of Authors’ Systems Per Metric,Camera View 1, Dataset:S2.L1, Time Sequence: 12.34.IEEE International Workshop on Performance Evalu-ation of Tracking and Surveillance, 2009.[7] J. Berclaz, A. Shahrokni, F. Fleuret, J. M. Ferryman,and P. Fua. Evaluation of probabilistic occupancy mappeople detection for surveillance systems. In EleventhIEEE International Workshop on Performance Eval-uation of Tracking and Surveillance, pages 55–62,2009.[8] D. Bolme, Y. Lui, B. Draper, and J. Beveridge. Sim-ple real-time human detection using a single correla-tion filter. In Twelfth IEEE International Workshop onPerformance Evaluation of Tracking and Surveillance,2009.[9] M. D. Breitenstein, F. Reichlin, B. Leibe, E. Koller-Meier, and L. van Gool. Markovian tracking-by-detection from a single, uncalibrated camera. InEleventh IEEE International Workshop on Perfor-mance Evaluation of Tracking and Surveillance, pages71–78, 2009.[10] A. B. Chan, M. Morrow, and N. Vasconcelos. Anal-ysis of crowded scenes using holistic properties. InEleventh IEEE International Workshop on Perfor-mance Evaluation of Tracking and Surveillance, pages101–108, 2009.[11] S. Choudri, J. Ferryman, and A. Badii. Robust back-ground model for pixel based people counting. InTwelfth IEEE International Workshop on PerformanceEvaluation of Tracking and Surveillance, 2009.[12] J.Ferryman and A. Shahrokni. An overview of thepets2009 dataset. In Twelfth IEEE International Work-shop on Performance Evaluation of Tracking andSurveillance, 2009.[13] R. Kasturi, D. Goldgof, P. Soundararajan, V. Manohar,J. Garofolo, R. Bowers, M. Boonstra, V. Korzhova,and Jing Zhang. Framework for performance evalu-ation of face, text, and vehicle detection and trackingin video: Data, metrics, and protocol. Pattern Analy-Figure 4: Median metric measurement across all views for different authors, Dataset:S2.L1, Time Sequence: 12.34.sis and Machine Intelligence, IEEE Transactions on,31(2):319–336, Feb. 2009.[14] N. Lehment, D. Arsic, A. Lyutskanov, B. Schuller, andG. Rigoll. Statistical filters for crowd image analysis.In Twelfth IEEE International Workshop on Perfor-mance Evaluation of Tracking and Surveillance, 2009.[15] P. K. Sharma, C. Huang, and R. Nevatia. Evaluationof people tracking, counting and density estimationin crowded environments. In Eleventh IEEE Interna-tional Workshop on Performance Evaluation of Track-ing and Surveillance, pages 39–46, 2009.[16] J. Yang, Z. Shi, P. Vela, and J. Teizer. Probabilisticmultiple people tracking through complex situations.In Eleventh IEEE International Workshop on Perfor-mance Evaluation of Tracking and Surveillance, pages79–86, 2009.Table 2: Labels and publication references for Figures 3, 4and 5Label ReferenceBazzani [5]Berclaz1 [7]Berclaz2 [7]Berclaz3 [7]Breitenstein [9]Leykin not publishedSharma [15]Yang [16]Arsic [14]ArsicWinter [4]Bolme ASEF [8]Bolme Cascade [8]Bolme Parts [8]Berclazdp [6]Berclazlp [6]Figure 5: Median metric per view per author.

PETS2009 and Winter-PETS 2009 results: a combined evaluation

Abstract

Similar works

Full text

Available Versions

Central Archive at the University of Reading