Designing evaluation methodologies: The case of motion detection

Motion detection is a fundamental processing step in the majority of visual surveillance algorithms. While an increasing number of authors are beginning to perform quantitative comparison of their algorithms, most do not address the complexity and range of the issues which underpin the design of good evaluation methodology. In this paper we explore the problems associated with optimising the operating point of detection algorithms and objective performance evaluation. A motivated and comprehensive comparative evaluation methodology is described and used to compare two motion detection algorithms reported in the literature. 1

An object-based comparative methodology for motion detection based on the F-Measure

The majority of visual surveillance algorithms rely on effective and accurate mo-tion detection. However, most evaluation techniques described in literature do not address the complexity and range of the issues which underpin the design of a good evaluation methodology. In this paper we explore the problems associated with both the optimising the operating point of any motion detection algorithms and the objective performance comparison of competing algorithms. In particular, we develop an object-based approach based on the F-Measure- a single-valued ROC-like measure which enables a straight-forward mechanism for both optimising and comparing motion detection algorithms. Despite the advantages over pixel-based ROC approaches, a number of important issues associated with parameterising the evaluation algorithm need to be addressed. The approach is illustrated by a com-parison of three motion detection algorithms including the well-known Stauffer and Grimson algorithm, based on results obtained on two datasets

A software for performance evaluation and comparison of people detection and tracking methods in video processing

WOS: 000294504600014Digital video content analysis is an important item for multimedia content-based indexing (MCBI), content-based video retrieval (CBVR) and visual surveillance systems. There are some frequently-used generic object detection and/or tracking (D&T) algorithms in the literature, such as Background Subtraction (BS), Continuously Adaptive Mean Shift (CMS), Optical Flow (OF) and etc. An important problem for performance evaluation is the absence of stable and flexible software for comparison of different algorithms. This software is able to compare them with the same metrics in real-time and at the same platform. In this paper, we have designed and implemented the software for the performance comparison and the evaluation of well-known video object D&T algorithms (for people D&T) at the same platform. The software works as an automatic and/or semi-automatic test environment in real-time, which uses the image and video processing essentials, e.g. morphological operations and filters, and ground-truth (GT) XML data files, charting/plotting capabilities and etc