Ekenel, H .K.

Rybok, L.

Stiefelhagen, Rainer

Voit, M.

English

Ekenel, H . K.

KITopen

Multi-view Based Estimation of Human Upper-Body OrientationLukas Rybok∗, Michael Voit†, Hazım Kemal Ekenel∗ and Rainer Stiefelhagen∗†∗ Institute of Anthropomatics,Karlsruhe Institute of Technology, Karlsruhe, GermanyEmail: {rybok, ekenel, rainer.stiefelhagen}@kit.edu† Interactive Analysis and Diagnosis,Fraunhofer IOSB, Karlsruhe, GermanyEmail: michael.voit@iosb.fraunhofer.deAbstract—The knowledge about the body orientation ofhumans can improve speed and performance of many servicecomponents of a smart-room. Since many of such componentsrun in parallel, an estimator to acquire this knowledge needsa very low computational complexity. In this paper we addressthese two points with a fast and efficient algorithm usingthe smart-room’s multiple camera output. The estimation isbased on silhouette information only and is performed foreach camera view separately. The single view results are fusedwithin a Bayesian filter framework. We evaluate our systemon a subset of videos from the CLEAR 2007 dataset [1] andachieve an average correct classification rate of 87.8 %, whilethe estimation itself just takes 12 ms when four cameras areused.Keywords-body-orientation; multi-view fusion; bag-of-featuresI. INTRODUCTION AND RELATED WORKIn recent years much research has been conducted to-wards enhancing the way people interact by unobtrusivelyproviding services in a smart-room environment. CHIL (http://chil.server.de) and AMI (http://amiproject.org) are just twoof the major European projects in this field. A typical smart-room is equipped with multiple acoustic and visual sensorswhich are used to analyze the scene in order to both providethe user contextual information and attempt to anticipate theuser’s needs. Therefore, an essential role in a smart-roomarchitecture is played by components providing means topercept human activities.The knowledge about the orientation of the body (anglearound the axis perpendicular to the ground plane) can bea useful cue. While tracking the position of people, thisinformation can help to deduce the most likely movementdirection of the target. Model-based human motion capturesystems can also benefit from this as the process of model-fitting is computationally very costly and the incorporationof the body orientation can reduce its complexity.Since in many cases the legs can be occluded by furnitureor other objects, it is beneficial to estimate the orientationof the upper-body only. The estimator should also yield avery low computational complexity to be useful because itwould be just one of many software components running inparallel. Thus, we present in this paper a fast approach toclassify the upper-body orientation of a person inside of asmart-room.To our best knowledge not much research has beenconducted to explicitly estimate the orientation of the humanbody. In [2] the estimation is based on the motion of atracked person and the size of its bounding ellipse. Ananalysis of the shape of the silhouette in images from ceilingmounted cameras is performed in [3], [4]. These approaches,though very fast, just give a course estimation and aresensitive to arm movement. Peng and Quian [5] use a multi-camera scenario and perform multi-linear analysis to extractorientation vectors from binary silhouette images. Fromthese a one-dimensional manifold is learned which is usedto retrieve the body orientation by solving a nonlinear leastsquares problem. However, a disadvantage of this approachis its low speed.In contrast to other works, our system is fast, accurate, andalso flexible in regard to the number of used camera views atthe same time. To be independent of the user’s appearance,we estimate the upper-body orientation using silhouetteinformation only. This approach however introduces am-biguities that are difficult to resolve using a monocularsetup (see Fig. 1). Therefore we show in our work that byincreasing the number of camera views such ambiguities canbe partly compensated and an overall increase in the systemperformance is achieved. In particular, we investigate theencoding of the silhouette information using either shapecontexts or histogram of shape context (HoSC) descriptorsas a base for the estimation. For each camera view in thesmart-room the orientation angle is classified separately andthe single hypotheses are fused within a Bayesian filterframework. This way we only need to train one estimatorthat can be used for any view as long as the inclinationangle of the camera to the ground plane is similar withthe views used for training. We evaluated our approach onfive sequences from the CLEAR 2007 evaluation datasetand achieved an average correct classification rate of 87.8%,when using 12 orientation classes.Figure 1. Ambiguity when only using silhouette based features: it isdifficult to tell appart a frontal view (left) from a rear view (right).II. ESTIMATING THE UPPER-BODY ORIENTATIONOur approach is based on silhouette features as they areeasy to extract by using foreground-background segmenta-tion, but at the same time contain much relevant informationto estimate the upper body orientation angle. For eachcamera view the estimation is carried out separately. Theupper-body region is extracted and the silhouette is encodedusing local features described in Sec. II-A. We then classifythe extracted feature vector to obtain a hypothesis for theorientation angle based on the observation of each cameraand fuse the results within a Bayesian filter framework asdescribed in Sec. II-B. Since we want to omit the needof training a separate classifier for each view, the outputof the classifiers is given relative to the coordinate systemof the capturing camera. The transformation to the worldcoordinate system is performed while merging the singlehypotheses.A. Image DescriptorsSince segmentation algorithms are not reliable enough toprovide noise free silhouettes, we need an encoding of thesilhouette information that is partly robust to segmentationfailures. The image descriptor should also be invariant toscaling as the input silhouettes are usually not of the samesize. Both shape contexts and HoSC descriptors providethese advantages.In [6] shape contexts are introduced as rich local descrip-tors of contour information. In order to transform a shapeto its shape context representation, the silhouette points aresampled, resulting in a set of n points. For each sampledpoint a shape context can be calculated as a histogram ofthe relative position of its neighboring points. The histogrambins are uniform in log-polar space making shape contextsmore sensitive to differences of nearby points. Invariance toscaling of the shapes is obtained by normalizing the pointdistances with the mean distance between all point pairs. Asthe histograms are calculated relative to each point, shapecontexts are also translation invariant.A major disadvantage of shape context descriptors is itshigh dimensionality. For instance, the parameters used inour experiments lead to 2400-dimensional feature vectors.A dimensionality reduction can be achieved using a bag-of-features scheme as described in [7]. The space spannedby all shape contexts obtained from the training images isfirst vector quantized using k-means clustering which yieldsa k-dimensional codebook of the cluster means. In orderto transform a new sample shape to a HoSC feature, itsshape contexts are binned to a histogram where each bin isassociated to one of the codebook vectors. Effects of spatialquantization are reduced by employing a soft-voting scheme.Thus we calculate the contribution η of a shape context ~SCto a cluster ~Ci as in [8] using:ηi( ~SC) =minj=1...k| ~SC − ~Cj |2| ~SC − ~Ci|2(1)The obtained ηi are summed up over all shape contexts andthe resulting histogram is normalized to unit length.B. Multi-view FusionSimilar to our previous work [9], we merge the estima-tions that are performed for each view to obtain the finalorientation class θ̂t by using Bayes’ theorem:θ̂t = arg maxxi∈Xc · p(Zt|xi)P (xi) (2)Here p(Zt|xi) denotes the class-conditional probability ofan observation Zt at time t given the i-th orientation classxi, with X = {xi} being the discrete state-space. P (xi) isthe prior probability of the occurrence of class xi and c isa normalizing constant being equal to the total probability.The class-conditional p(Zt|xi) is calculated by averagingthe classification confidence for each view, which is basedon a confusion matrix learned on validation data. The priorP (xi) depends on the previous state x′ at time t− 1, as theturning speed of the body is usually limited. Thus it can becomputed as:P (xi) =∑x′∈XP (xi|x′)P (x′|Zt−1), (3)where the state-transition probability P (xi|x′) we modelwith a zero-mean Gaussian N0;σ(xi − x′). We determinedthe standard deviation σ during experiments on validationdata and set it to 20◦. When no preceding state is known,we assume the prior to be equal for each state.III. EXPERIMENTSWe evaluated the proposed approach on a subset of videosequences from the CLEAR 2007 head-pose evaluationdataset. Each of the sequences consists of about 2700 framesand contains one person that randomly changes his bodyorientation. The camera setup used to capture the videoscan be seen in Fig. 2. Within the sequences we annotatedthe position of the shoulders in all views, so that the bodyFigure 2. Camera setup of the smart-roomposition and orientation could be retrieved using multi-viewgeometry. Additionally, we segmented foreground regionsusing the algorithm presented in [10]. Because the annota-tion process is very time-consuming, our test set consistedof five 3 minutes long sequences (15 fps, 320×240 pixels),each containing one different person.The evaluation was performed using a leave-one-out ap-proach, where each sequence was used as testing data once,while the remaining sequences were used for training. Inour experiments we evaluated SVM classification togetherwith HoSC features (hosc svm) and Nearest-Mean classifi-cation (NMC) with both shape contexts (sc nmc) and HoSCfeatures (hosc nmc). Because of the high dimensionality ofshape context features and limited amount of training data,we did not further investigate SVM classification of shapecontexts in our experiments. This was also supported by verylow initial experimental results. We favor the use of NMCover Nearest-Neighbor classification, since the low speed ofthe latter one prohibits its use in a smart-room environment.For the SVM classification we use a RBF kernel and aone-vs-one approach. We handle the multi-class case witha max-win voting strategy, where each SVM votes forone class and the recognition result is the mostly votedclass. Both classifiers use the χ2 metric as a distancefunction between features. When encoding a silhouette withshape contexts, all extracted features from one silhouetteare concatenated to one feature vector. We determined allparameters experimentally on separate validation data andset the histogram dimensions to r = 6 and a = 8 forHoSC features and r = 4 and a = 8 for shape contexts. Wesampled 50 points equidistantly on the upper-body silhouetteand used a 50 element HoSC codebook. The size of one classwas set to 30◦ since it makes the evaluation independent oferrors introduced through the labeling process.In our first experiment we used an all-to-all classificationscheme where only one classifier is trained using data fromall four views and used to estimate the orientation angle inTable IOVERALL PERFORMANCE OF THE ALL-TO-ALL CLASSIFICATIONSCHEMEMethod Correct Classhosc svm 82.6 %hosc nmc 79.1 %sc nmc 85.0 %Table IIOVERALL PERFORMANCE OF THE ONE-TO-ONE CLASSIFICATIONSCHEMEMethod Correct Classhosc svm 65.9 %hosc nmc 76.8 %sc nmc 87.8 %each view. As can be seen in Tab. I, sc nmc outperformsthe other approaches with an average correct classificationrate of 85.0%.In a multi-camera setup, the body silhouette my lookdifferent depending on the distance from the camera. Inorder to investigate the impact of such appearance changeson the performance of our approach, we used in the nextexperiment for each camera a separate classifier trained onlywith data captured with that camera (one-to-one scheme).In Tab. II it can be observed that the performance of theNMC approach deviates only insignificantly from the resultsin the previous experiment showing its view-independence.However, when using the SVM classifier a clear decreasein performance is visible which we believe is due to thesmaller amount of training data per estimator caused by theexperimental setup.In a final experiment, we show the benefits of using amulti-camera setup to estimate the orientation angle. To dothis, we evaluated the system performance when using allpossible combinations of camera views during testing andaveraging the results for each number of views that are usedto derive the final estimation. In Fig. 3, it can be clearlyseen that with each additional view the performance of allsystems improves greatly. When comparing the results of asingle camera based estimation with the results of using allfour views, the highest relative performance gain of 38% canbe observed for sc nmc. These results strongly suggest thatbasing the estimation on multiple views can compensate forthe loss of discrimination between frontal and rear views.Using an unoptimized C++ implementation on a 3GHzIntel Pentium IV machine, the computation of one con-catenated shape context feature vector takes 2ms and itsclassification with the NMC additional 1ms. This clearlyshows the suitability of the proposed approach as an auxil-iary component in a smart-room environment.Figure 3. Overall performance (one-to-all scheme) when changing thenumber of viewsIV. CONCLUSIONWe presented a classification based approach to estimatethe upper-body orientation of a person inside of a smart-room. For each of the camera views used in the sensorsetup of the room, we build a single hypothesis of theorientation angle using silhouette information only. A jointmeasurement is then obtained with a Bayesian filter frame-work. This makes our approach flexible, as no retraining isneeded whenever the camera setup is changed. We showedthat the use of four cameras instead of a single one greatlyimproves the performance of our approach and achieved anaverage correct classification rate of up to 87.8% on fivesequences from the CLEAR 2007 evaluation dataset. Thehigh performance and low computational complexity (12msper estimation when using four views) makes our approachwell suitable as an auxiliary component in a smart-roomscenario.ACKNOWLEDGMENTThis study is partially funded by the German ResearchFoundation (DFG) under Sonderforschungsbereich SFB 588- Humanoid Robots - and by BMBF, German FederalMinistry of Education and Research as part of the GEMSprogramme.REFERENCES[1] R. Stiefelhagen, K. Bernardin, R. Bowers, R. T. Rose,M. Michel, and J. Garofolo, “The CLEAR 2007 evaluation,”CLEAR 2007 and RT 2007, pp. 3–34, 2008.[2] M. W. Lee and R. Nevatia, “Body part detection for humanpose estimation and tracking,” in IEEE Workshop on Motionand Video Computing, 2007, p. 23.[3] S. Iwasawa, J. Ohya, K. Takahashi, T. Sakaguchi, K. Ebihara,and S. Morishima, “Human body postures from trinocularcamera images,” in Intl. Conf. on Automatic Face and GestureRecognition, 2000, pp. 326–331.[4] W. Zhang, T. Matsumoto, J. Liu, M. Chu, and B. Begole,“An intelligent fitting room using multi-camera perception,”in Intl. Conf. on Intelligent User Interfaces, 2008, pp. 60–69.[5] B. Peng and G. Qian, “Binocular dance pose recognition andbody orientation estimation via multilinear analysis,” in CVPRWorkshops 2008, 2008, pp. 1–8.[6] G. Mori, S. Belongie, and J. Malik, “Shape contexts enableefficient retrieval of similar shapes,” in CVPR 2001, vol. 1,2001, pp. 723–730.[7] A. Agarwal and B. Triggs, “3D human pose from silhouettesby relevance vector regression,” in CVPR, 2004.[8] R. Poppe and M. Poel, “Comparison of silhouette shapedescriptors for example-based human pose recovery,” in Intl.Conf. on Automatic Face and Gesture Recognition, 2006, pp.541–546.[9] M. Voit, K. Nickel, and R. Stiefelhagen, “A bayesian approachfor multi-view head pose estimation,” in IEEE MFI 2006,2006, pp. 31–34.[10] Z. Zivkovic and F. van der Heijden, “Efficient adaptivedensity estimation per image pixel for the task of backgroundsubtraction,” Pattern Recognition Letters, vol. 27, no. 7, pp.773–780, 2006.

Multi-view Based Estimation of Human Upper-Body Orientation

https://publikationen.bibliothek.kit.edu/1000026385/2024011

Multi-view Based Estimation of Human Upper-Body Orientation

Abstract

Similar works

Full text

Available Versions

KITopen