Video face matching using subset selection and clustering of probabilistic multi-region histograms

Balancing computational eciency with recognition accuracy is one of the major challenges in real-world video-based face recognition. A signicant design decision for any such system is whether to process and use all possible faces detected over the video frames, or whether to select only a few `best' faces. This paper presents a video face recognition system based on probabilistic Multi-Region Histograms to characterise performance trade-os in: (i) selecting a subset of faces compared to using all faces, and (ii) combining information from all faces via clustering. Three face selection metrics are evaluated for choosing a subset: face detection condence, random subset, and sequential selection. Experiments on the recently introduced MOBIO dataset indicate that the usage of all faces through clustering always outperformed selecting only a subset of faces. The experiments also show that the face selection metric based on face detection condence generally provides better recognition performance than random or sequential sampling. Moreover, the optimal number of faces varies drastically across selection metric and subsets of MOBIO. Given the trade-os between computational eort, recognition accuracy and robustness, it is recommended that face feature clustering would be most advantageous in batch processing (particularly for video-based watchlists), whereas face selection methods should be limited to applications with signicant computational restrictions