Heizmann, M.

Laubenheimer, A.

Wetzel, J.

Zeitvogel, S.

English

Crossref

Towards Global People Detection and Tracking using Multiple Depth Sensors

KITopen

Towards Global People Detection and Trackingusing Multiple Depth SensorsJohannes Wetzel, Samuel Zeitvogel, Astrid LaubenheimerIntelligent Systems Research Group (ISRG)Karlsruhe University of Applied SciencesKarlsruhe, Germany{johannes.wetzel,samuel.zeitvogel,astrid.laubenheimer}@hs-karlsruhe.deMichael HeizmannInstitute of Industrial Information Technology (IIIT)Karlsruhe Institute of Technology (KIT)Karlsruhe, Germanymichael.heizmann@kit.edu©2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, includingreprinting/republishing this material for advertising or promotional purposes,creating new collective works, for resale or redistribution to servers or lists, orreuse of any copyrighted component of this work in other works. DOI: 0.1109/ISETC.2018.8583962Abstract—In this work a novel approach for multi depth sensorperson detection and tracking from top view is presented. Wepropose a probabilistic framework formulating the problem ofpeople detection in multiple overlapping depth images as aninverse problem. As a generative forward model, we employa simple differentiable 3D person model allowing us to detectpeople from arbitrary viewpoints. Furthermore, we extend ourprobabilistic framework to allow for tracking of individualsover time. Finally, we show how to solve for the global persontrajectories exploiting differentiable rendering. The preliminaryevaluation shows promising qualitative results of our approachon samples of three stereo vision based depth sensors observingan indoor scene.Index Terms—multi camera person detection and tracking;multi sensor fusion; network of depth cameras; inverse graphics;inverse problem; generative model; differentiable renderingI. INTRODUCTIONTracking and detection of people in a network of cameras isa vital preprocessing task for many applications such as videosurveillance, sport game analysis, ambient assisted living, etc.The vast majority of the related literature focuses on theclassical video surveillance scenario, capturing the pedestriansfrom profile and frontal view using 2D video cameras. In thiswork we focus on a different setup. We address the problemof indoor people detection and tracking using a network ofstereo vision based depth sensors. In contrast to the classicalvideo surveillance scenario, the sensors capture the scene froma top view to resolve occlusion in crowded scenes. Due todrastic view point change the persons strongly vary in theirappearance, making it very challenging for classical discrimi-native pedestrian detectors. Moreover, the majority of existingliterature of multi camera tracking relies on a local detectionapproach, where the local detections from every sensor aremerged into a global coordinate system. Instead, we formulatethe problem of people detection and tracking in multiple depthimages as inverse problem, seeking to overcome the challengesmentioned above in two ways:1) Generative person model: In contrast to discriminativepedestrian detection algorithms we employ a generativeThe authors would like to thank the German Federal Ministry of Educationand Research (BMBF), for funding the presented research under grant#13FH025IX6.3D person model, leading to a view-point independentdetector which does not depend on a huge training set.2) Global detection and tracking: Our approach modelsthe detection and tracking with a global probabilitydistribution conditioned on all observations at every timestep. Thus, the given dependency between all cameraviews can be exploited in a natural way. Our approachis capable of integrating redundant as well as comple-mentary information from different views, attempting toresolve occlusion as well as measurement noise.II. RELATED WORKMulti camera people detection and tracking has been widelystudied in the context of video surveillance. The vast majorityof those approaches are based on multiple 2D video camerasobserving an outdoor scene. However the topic of peopledetection and tracking with a network of depth cameras,especially in top down view, is not well studied yet. Thereforewe will first discuss the different methods of the classicalvideo surveillance literature and later focus on the morespecialized depth sensor setup we address in this work. Thetask of tracking people in a multi camera network can bedivided into approaches working across non-overlapping views[1] and overlapping views [2]. In this work we will onlyfocus on approaches working across overlapping views. Fora comprehensive review of multi camera people tracking werefer to [3], [4]. For the rest of this section we categorize theliterature on multi camera person tracking in approaches basedon local detection and tracking, homography based approachesand inverse problem approaches.Since detecting and tracking people in a single cameraimage has been intensivly studied [5], [6] a lot of approachesrely on fusing local detections or local tracklets into a commoncoordinate system. The fusion of the tracklets of differentcameras can be achieved by spatio temporal features [7] andappearance based features [8]. Since each camera is detectingpeople independently, those approaches do not make full useof the given multi view information, making it very difficultto handle occlusion. Moreover, the applied standard peopledetectors are optimized to detect pedestrians in frontal viewand profile view but not in the top view [9] making theminsufficient for our setup.Homography based approaches project local image cues,e.g. single points of interest, pixel intensities or the silhouetteof a blob on a common plane and fuse this representationacross all views to get global detections [10]. Eshel et al. [11]project the foreground pixels of all views into common heightplanes to detect the heads of people. Kahn et al. [2] focuseson the region around the foot point of people proposing thehomographic occupancy constraint to handle occlusion. Penget al. [12] extends those approaches by a multi view Bayesiannetwork to avoid ”phantom” detections due to heavy occlusion.However, all mentioned approaches rely heavily on a good 2Dforeground segmentation, hence suffering from noise, shadowsand illumination changes.A more generic class of approaches formulates the problemof multi camera people detection as an inverse problem by em-ploying a generative person model and minimize the differencebetween the image evidence of all cameras and the syntheticimages. Fleuret et al. [13] propose a probabilistic frameworkfor multi camera people detection and tracking using a simplegenerative person model in form of a rectangular boundingbox. Alahi et al. [14] present a similar approach using a morecomplex silhouette as a generative forward model. In contrastto our method, both approaches employ only 2D forwardmodels and fit the model to a binary foreground mask.Tseng et al. [15] present an indoor surveillance system,based on multiple top view kinect depth cameras. For eachcamera they obtain a virtual top view depth image based onthe given point cloud, finally stitching a global depth image.Moreover, they apply a hemiellipsoidal head model to detectedpeople in the global depth image. However, the approach relieson high quality depth data and is limited to the top view.The present work is inspired by [13] but in contrast we areusing depth images as evidence and propose a differentiablegenerative 3D model using OpenDR [16], allowing us toeffectively detect people in arbitrary viewpoints. To the best ofour knowledge, the problem of people detection and trackingin multiple top view depth images has not yet been formulatedas inverse problem using a differentiable generative 3D model.III. APPROACHWe formulate the people detection and tracking problem asan inverse problem using a generative 3D person model. In theliterature this method is also referred to as vision-as-inverse-graphics or analysis-by-synthesis. We assume that the sensorsare intrinsically and extrinsically calibrated in advance. LetPc = Kc[Rc|~tc] be the projection matrix for each camerac, with Kc being the intrinsic camera matrix and [Rc|~tc]the extrinsic transformation which maps a point from thecommon world coordinate system to the corresponding cameracoordinate system. For convenience, we assume that we knowthe number of people n in the scene a priori. In a practicalsystem, this limitation can be overcome by applying a 2Ddetector or make use of an iterative scheme trying to find thetrue value for n. For better understanding, we will first discussthe detection framework for a single time step and afterwardsextend it, incorporating a sequence of frames.Let n be the number of people in the scene and ~X =(~x1, . . . , ~xn) be the vector describing the person locations~x ∈ R2 in ground plane world coordinates. Our goal is toinfer how probable a scene configuration ~X explains the givenobservations ~O = (O1, . . . , OC) from C cameras at the sametime step. In this work we use depth images as observations.Applying Bayes’ theorem we get the posterior distributionp( ~X| ~O) = p(~O| ~X)p( ~X)p( ~O). (1)A. LikelihoodWe assume that, the views are independent for a fixedconfiguration ~X . Thus, the likelihood factorizes as follows:p(O1, ..., OC | ~X) =C∏c=1p(Oc| ~X). (2)We model the likelihood using a generative forward modelG(·) which maps a scene configuration ~X and a given projec-tion matrix Pc to a synthetic observation (i.e. synthetic depthimage) from camera c using a simple 3D person model. Themodel consists of a cylinder for the body and a sphere for thehead, see Fig.1(c). For rendering, we use the differentiablerenderer OpenDR [16]. Assuming that our given observationssuffer from Gaussian noise, we can formulate the likelihoodasp(Oc| ~X, σ) ∝ exp(− 12σ2||Oc −G( ~X,Pc)||2). (3)We incorporate the physical model of the sensor in a naturalway in to our framework, allowing us to detect people fromarbitrary viewpoints and to easily integrate a new sensormodality into the network.B. PriorFor the detection we employ two independent priors,pbox( ~X) and pdist( ~X), hence p( ~X) = pbox( ~X)pdist( ~X). Sincewe can estimate the visible ground plane of the sensor net-work, we model this knowledge as a uniform distributed priorfor every person location in the observable rectangular area,thus we can writepbox( ~X) =n∏i=1p(~xi), p(~xi) = U(~xmin, ~xmax) (4)with U(~xmin, ~xmax) being the uniform distribution overthe approximated rectangular area given by the interval[~xmin, ~xmax]. The second prior exploits the assumption thattwo individuals are keeping a certain distance to each other.Since we assume that the probability pdist( ~X) depends only onthe joint probabilities of all possible location pairs, we modelthe second prior aspdist(~xi, . . . , ~xn) =n−1∏i=1n∏j=i+1p(~xi, ~xj). (5)Modelling the joint probability between two locations as a zeromean Gaussian with respect to the inverse distance d(~xi, ~xj) =(a)(b)(c)Fig. 1. Example from an indoor sequence with three sensors. A column corresponds to one sensor, row (a) shows the camera images with reprojected footpoints of the inferred detections, (b) shows the observed depth images after background subtraction, used as input for our approach, (c) shows the correspondingsynthetic depth images.1/(||~xi− ~xj ||+ ε) we can write the pairwise joint probabilitydistribution asp(~xi, ~xj |σ) ∝ exp(− 12σ2||d(~xi, ~xj)||2). (6)C. Maximum a posteriori estimationTo find the maximum a posteriori (MAP) scene configura-tion as defined in (1) we solve the following non-linear least-squares problem:~X∗ = argmax ~X p(~X|O1, . . . , Oc)= argmin ~XC∑c=1||Oc −G( ~X,Pc)||2 + αEbox + βEdist(7)We approximate the box prior in (4) in a way that is compu-tational preferable for continuous numerical optimization. Letxi,r be r-th component of ~xi, then the box penalty is given asEbox =n∑i=1∑r∈{1,2}[max(xmin,r − xi,r, 0)+ max(xi,r − xmax,r, 0)]2.(8)The distance energy term is a direct result of the prior proposedin (5), with an additional max function, which makes sure thatthe costs are zero for all pairwise distances larger than δ = 1m:Edist =n−1∑i=1n∑j=i+1[max(d(~xi, ~xj)− δ−1, 0)]2. (9)The final non linear least squares problem is solved usinga trust region method like dogleg, exploiting the gradientsprovided by the differentiable renderer framework OpenDR[16]. For better convergence we apply a coarse to fine strategyusing a Gaussian image pyramid.D. TrackingSo far, we have discussed inferring the locations ~Xt ofpeople given the observation of all cameras for one time step t.Given a time series of m observations O = ( ~O1, . . . , ~Om) wecan extend the framework defined above to infer the locationof every individual at every time step X = ( ~X1, . . . , ~Xm).We rewrite the posterior distribution in (1) asp(X|O) =[∏t∏c p(Otc| ~Xt)]p(X )p(O). (10)We employ an additional prior ptime(X ) which represents thedynamics between consecutive observations. Since we assumeonly small movements between two frames, we use a simpleMarkov modelptime(X ) =m∏t=2n∏i=1p(~xti|~xt−1i ) (11)assuming that a person location at time step ~xt is just a noisyversion of its predecessor ~xt−1, thus we can writep(~xti|~xt−1i ,Σ) = N (~xt−1i ,Σ). (12)Since the likelihood and prior terms defined in (2) and (4,5)respectively are staying the same for every single time step,we only need to sum those up over time and add the additionalenergy term for the prior ptime(X ). The total MAP objectivefor global detection and tracking is then given asX ∗ = argminXm∑t=1C∑c=1||Otc −G( ~Xt,Pc)||2+αm∑t=1Ebox + βm∑t=1Edist + γm∑t=2|| ~Xt − ~Xt−1||2.(13)IV. EVALUATIONDue to the lack of a publicly available dataset for multidepth sensor people detection and tracking in top view, wepresent preliminary qualitative experiments, showing the ap-plicability of our approach. We evaluate our approach on anindoor office scene recorded from three stereo vision baseddepth sensors. The sensors have a top view on the scene andare mounted at a height of three meters, having a significantoverlap to each other, see Fig. 1(a). As input observations weuse foreground depth images in a resolution of 376 × 240,obtained by static background subtraction, see Fig. 1(b). Weuse α = 0.1, β = 0.01, γ = 0.01 as weight parameters for theregularization terms in (13).Fig. 1 illustrates the MAP solution for one sample frame.Our generative model (Fig. 1(c)) is able to explain the givenobservations (Fig. 1(b)) quite well. Notice how our approachmakes use of all image evidence, even if people are onlypartially visible. This improves the global detection result. InFig. 2 we use the image evidence of two sensors only. Peopleare correctly located, even when heavy occlusion is present(Fig. 2(a)) and the viewpoint is quite extreme (Fig. 2(b)).V. CONCLUSIONIn the present work we have addressed the problem ofpeople detection and tracking in multiple overlapping depthimages as an inverse problem. We have proposed a naturalprobabilistic formulation for people detection and tracking,using a generative 3D person model. Moreover, we haveshown how to solve for the global MAP solution, exploitingdifferentiable rendering.Future work will include a quantitative evaluation of theproposed approach as well as the investigation of approximatebayesian inference methods such as Markov chain Monte(a) (b)(c) (d)Fig. 2. Example with two sensors. A column corresponds to one sensor. Fig.(a) and (b) show the camera images, (c) and (d) the observed depth images.Carlo or variational inference to not only approximate thefull posterior distribution but also overcome the shortcomingsof the proposed gradient based optimization strategies, e.g. thea priori needed number of persons in the scene.REFERENCES[1] A. Rahimi, B. Dunagan, and T. Darrell, “Simultaneous calibration andtracking with a network of non-overlapping sensors,” in CVPR, vol. 1,pp. 187–194, 2004.[2] S. M. Khan and M. Shah, “Tracking multiple occluding people bylocalizing on multiple scene planes,” TPAMI, vol. 31, no. 3, 2009.[3] X. Wang, “Intelligent multi-camera video surveillance: A review,” Pat-tern Recognition Letters, vol. 34, pp. 3–19, jan 2013.[4] L. Hou, W. Wan, J.-N. Hwang, R. Muhammad, M. Yang, and K. Han,“Human tracking over camera networks: a review,” EURASIP Journalon Advances in Signal Processing, no. 1, p. 43, 2017.[5] S. Zhang, R. Benenson, M. Omran, J. Hosang, and B. Schiele, “To-wards Reaching Human Performance in Pedestrian Detection,” TPAMI,vol. 8828, no. c, pp. 1–1, 2017.[6] H. Kieritz, S. Becker, W. Hubner, and M. Arens, “Online multi-persontracking using Integral Channel Features,” AVSS, pp. 122–130, 2016.[7] N. Anjum and A. Cavallaro, “Trajectory association and fusion acrosspartially overlapping cameras,” AVSS, pp. 201–206, 2009.[8] G. Kayumbi, N. Anjum, and A. Cavallaro, “Global trajectory recon-struction from distributed visual sensors,” ACM/IEEE InternationalConference on Distributed Smart Cameras, ICDSC, 2008.[9] C. Ertler, H. Possegger, M. Opitz, and H. Bischof, “Pedestrian Detectionin RGB-D Images from an Elevated Viewpoint,” in 22nd ComputerVision Winter Workshop, CVWW, 2017.[10] B. A. C. Sankaranarayanan, A. Veeraraghavan, and R. Chellappa, “Ob-ject Detection, Tracking and Recognition for Multiple Smart Cameras,”Proceedings of the IEEE, vol. 96, no. 10, 2008.[11] R. Eshel and Y. Moses, “Homography based multiple camera detectionand tracking of people in a dense crowd,” CVPR, 2008.[12] P. Peng, Y. Tian, Y. Wang, J. Li, and T. Huang, “Robust multiple cam-eras pedestrian detection with multi-view Bayesian network,” PatternRecognition, vol. 48, no. 5, pp. 1760–1772, 2015.[13] F. Fleuret, J. Berclaz, R. Lengagne, and P. Fua, “Multicamera peopletracking with a probabilistic occupancy map,” TPAMI, vol. 30, 2008.[14] A. Alahi, L. Jacques, Y. Boursier, and P. Vandergheynst, “Sparsity drivenpeople localization with a heterogeneous network of cameras,” Journalof Mathematical Imaging and Vision, vol. 41, no. 1-2, pp. 39–58, 2011.[15] T. E. Tseng, A. S. Liu, P. H. Hsiao, C. M. Huang, and L. C. Fu, “Real-time people detection and tracking for indoor surveillance using multipletop-view depth cameras,” IROS, pp. 4077–4082, 2014.[16] M. M. Loper and M. J. Black, “OpenDR: An approximate differentiablerenderer,” ECCV, vol. 8695 LNCS, no. PART 7, pp. 154–169, 2014.

https://publikationen.bibliothek.kit.edu/1000091197/72168182

Towards Global People Detection and Tracking using Multiple Depth Sensors

Abstract

Similar works

Full text

Available Versions

Crossref

KITopen