In this paper, we address the problem of reconstructing 3D volumetric models, illustrating human sporting performance for use in coaching scenarios. We advocate the use of low cost camera networks for acquiring such data, allowing the approach to be feasibly adopted by both amateur and elite level sports athletes. A dynamic voxel carving approach is described, coupled with over-head player tracking and autonomous background subtraction, to automatically produce a 3D reconstruction technique that intelligently uses memory resources. We demonstrate the efficacy of our approach in the context of tennis as a challenging application scenario

Kelly, Philip

Monaghan, David

O'Connor, Noel E.

DCU Online Research Access Service

DYNAMIC VOXEL CARVING IN TENNIS BASED ON PLAYER LOCALISATION USING ALOW COST CAMERA NETWORKDavid Monaghan, Philip Kelly and Noel E. O’ConnorCLARITY: Centre for Sensor Web Technologies, Dublin City University, IrelandABSTRACTIn this paper, we address the problem of reconstructing 3Dvolumetric models, illustrating human sporting performancefor use in coaching scenarios. We advocate the use of lowcost camera networks for acquiring such data, allowing theapproach to be feasibly adopted by both amateur and elitelevel sports athletes. A dynamic voxel carving approach isdescribed, coupled with over-head player tracking and au-tonomous background subtraction, to automatically producea 3D reconstruction technique that intelligently uses memoryresources. We demonstrate the efficacy of our approach in thecontext of tennis as a challenging application scenario.Index Terms— Image processing, Reconstruction algo-rithms, Voxel carving1. INTRODUCTIONIn order for a coach to improve an athlete’s performance, bothtechnically and tactically, deficiencies in the student’s abil-ities must be both identified and eliminated – starting withlarger faults in technique and tactical awareness, and finallyfine tuning to remove smaller issues. However, these twocomplementary objectives of identification and eliminationare difficult to achieve in practice. Identifying the factorsneeding improvement, requires a solid understanding of themany variables critical to optimal performance. In addition,the coach must be able to effectively convey this informationto the player for the benefits of the coach’s knowledge to beefficiently leveraged.In this paper, we describe a non-invasive approach for ac-quiring a 3D reconstruction of a human’s performance usinga low cost camera network. An accurate 3D rendering ofan athlete’s motion can be used to maximise the impact ofcoaching feedback, allowing a coach further flexibility whenclarifying a tactical issue, by allowing the game to be viewedfrom any angle. In addition, the coach has the ability to re-play the match from the point of view of either player. Thiscan provide the athlete with highly beneficial tactical infor-mation obtained from reliving the match not only from theirown viewpoint, or a spectator’s viewpoint, but also from theiropponent’s point of view.This paper is organized as follows: Section 2 details pre-vious work in the area. Section 3 provides a description of the3D reconstruction of an athlete’s movements from a numberof low cost cameras. In section 4 we quantitatively evaluateour approach in a real world test scenario, namely for obtain-ing rapid motions from a tennis player in competitive action.Tennis is a challenging test scenario, given both the speed andexplosive nature of the actions performed by athletes. Finally,section 5 provides conclusions and future work.2. PRIORWORKThe main motivation in this work is to extract 3D motion fromathletes using low cost techniques, which can be feasibly ac-quired by most amateur sporting clubs. This data can thenbe applied in performance evaluation from both training andmatch scenarios. Although sensor-based techniques could beemployed in training, such as the strapping of wireless wear-able sensors [1] to players, the use of such technologies incompetitive action is currently not a feasible solution, due tosporting body regulations. In addition, despite the relativelysmall size of such sensors, discussions with elite athletes andsports professionals lead us to believe that their size could stillimpair the performance of athletes. As such, in this work wefocus on the use of non-invasive, image processing techniquesfrom camera networks.Space carving, or voxel (three dimensional pixel), tech-niques [2, 3, 4, 5] have received much attention in the litera-ture in recent years. These approaches create a 3D represen-tation of a subject from multiple 2D images, synchronouslycaptured from a number of calibrated cameras at differentviewing angles. In each camera view, the subject of interestis identified, and its silhouette is segmented from the imagebackground. The approach of space carving, as described inthe seminal work of Kutulakos and Seitz [4], is to progres-sively remove inconsistent voxels from a pre-defined initialvolume, using the extracted silhouettes from each image. Theoutput of the technique is the maximum volume in 3D spacewhich may have produced the extracted silhouettes. However,the choice of this initial volume can have a considerable im-pact on the outcome of the reconstruction process [4].Many others have expanded on the Kutulakos and Seitzalgorithm. Broadhurst and colleagues present a probabilis-tic approach to space carving, accessing the likelihoods of avoxel existing or not existing [2]. This approach benefits fromassuring no holes are carved in the final model. A more ac-curate carving, in a moving environment, is produced by in-troducing a ’hexel’ and performing motion carving in six di-mensions in [6]. Yang at el. presents a progressive scheme toimprove the reconstruction of surfaces that lack sufficient tex-ture details coupled with robustness to variations in lightingconditions [7]. Kolmogorov and Zabih take a slightly differ-ent approach, and employ an energy minimization techniquevia graph cuts [8]. Bhatia at el. designed a bottom-up 3Dlimb detection system that utilises a multi-view Eigen modelto provide approximate 3D pose information that combinesthe concepts of space carving and Eigen recognition [9]. Seitzand colleagues presented a quantitative comparison of sev-eral multi-view stereo reconstruction algorithms and try to ad-dress the challenges in comparing and evaluating multi-viewstereo algorithms [5]. Real time space carving implementa-tions have been proposed that use graphics hardware [10] orseveral computers [11]. Cheung at el. [11] present a near-realtime system for voxel reconstruction that uses multiple com-puters, one computer is used for each camera in the systemand a further computer to compute the final voxel carving.3. APPROACHIn general, there are a number aspects required to achieve highquality voxel carvings; (a) a high number of images, capturedfrom multiple calibrated cameras at varying viewing angles;(b) good camera calibration; (c) quality object silhouette seg-mentation; and (d) the initial volume selected for carving.Within these constraints, the number of images available isgenerally predetermined by the number of available camerasin a particular physical setup, and good manual calibrationcan ensure the second.In this work, we focus on the last two aspects, extend-ing the work of Yang at el. [12] to cope with the difficulties inobject segmentation in relatively unconstrained real world en-vironments. In addition, we detail a technique for providinga good choice for the 3D initial volume to be used for carv-ing. The camera setup used in this study, shown in figure 1(a),consists of nine low cost IP-network cameras, (seven cameraswith 640 × 480 pixels and two cameras with 704 × 576 pix-els), forming a 220 degree arch around a tennis court and oneoverhead camera. This overhead camera was a specific re-quirement of the tennis coaches, in order to visualise tacticalshots and movement during matches. It should be noted thatin this scenario, 360 degree ring of cameras was not possiblein this case due the presence of three additional tennis courtsbeside the test court. However this represents a realistic courtsetup that could be obtained in most amateur sports clubs.This camera setup tends to differ significantly when com-pared to those adopted in the voxel carving, both in termsof camera pixel resolution and the spatial volume that recon-(a)(b) (c)Fig. 1. (a) Camera Setup; (b/c) Player tracking.struction can occur (the nine cameras cover the 78 × 36 feetsize of a standard tennis court, in addition the overhead cam-era is 45 feet above the ground). In our experimental setup,the spatial region covered by a bounding box of a human sub-ject can range from maximum of 92 × 357 pixels (width ×height) when the athlete is close to a given camera, to a min-imum of 48 × 135 pixels when the player is at the opposingend of the court to the camera. In addition, the spatial regioncovered by the player from the overhead viewpoint is an aver-age of 42× 21 pixels. As we are focusing on low cost cameranetworks, the camera pixel resolution and number of camerasshould be kept low, in order to minimise expense. As such,it is imperative that the segmentation quality of the silhou-ette must be high. For each camera viewpoint, we employ alayered background modelling approach, which includes ro-bust shadow removal techniques [1], to learn the backgroundscene. However, even with such a technique, accurate seg-mentation is difficult to achieve in real world scenarios, wheremoving backgrounds, lighting conditions and lack of variabil-ity in colour intensity between players and background ob-jects in the scene are unconstrained. In order to maximisethe accuracy of the segmentation technique, we leverage thebenefits of having multiple calibrated cameras.An overhead camera is used to perform player tracking[13], which identifies the groundplane player location on thecourt in real-world Euclidean coordinates (see figure 2(b),where the location of the base of the cube indicates the playerposition). A player-cube is then placed around the spatial vol-(1)(2)(3)(4)(a) (b) (c) (d)Fig. 2. Player localisation and silhouette extraction.ume occupied by the player. The cube’s groundplane dimen-sions are determined using the bounding box of the playertracker, the height of the cube is set to 2.4 metres (see fig-ure 1(c)). As the intrinsic and extrinsic camera calibrationparameters of each the other eight cameras in the networkare known, the player-cube location can be obtained in eachone of the other camera’s local coordinate systems – see fig-ure 2 row (2), for the projection of a 3D player-cube ontofour different camera image planes. Using the bounds of theplayer-cube on each image plane, the player location can beconstrained to a specific section of the image. As such, inorder to segment the players silhouette from each viewpoint,foreground pixels are obtained (using the layered backgroundmodelling technique of O´ Conaire and colleagues[1]) fromwithin the bounds of the 2D player-cube only. These pix-els are then filtered to remove noise, via connected compo-nent analysis and regions under a predefined threshold are re-moved. This technique of silhouette extraction can cut downon computational complexity, while simultaneously increas-ing precision on silhouette extraction – see figure 2 rows (2)and (3) for example results of this process.Once player silhouettes are extracted, space carving as perthe technique of Yang at el. [12] can be performed. However,the amount of computer memory utilised in traditional spacecarving techniques can be both computationally and mem-ory intensive. In order to reduce this complexity, the num-ber of voxels used in the initial volume selected for carvingcan be decreased. This technique will also reduce the spa-tial resolution of the 3D volumetric reconstruction. In thiswork, we use the available computer memory as efficientlyas possible by adopting a dynamic space carving methodol-ogy, which is controlled by player localisation on the court,again provided by the overhead camera tracker. The initialvolume used for carving is centred, as before, on the area(a) (b) (c)(d) (e) (f)Fig. 3. Volumetric reconstructions using (a) 0, (b) 1, (c) 2, (d)3 and (e) 4 camera viewpoints. Figure (f) shows the resultantvolumetric model with surface colouring.occupied by the player. Only this specific volume is filledwith voxels. As such, we can keep the spatial resolution ofthe voxels constant, but the number of voxels can be dynam-ically changed with respect the size of the player boundingbox supplied by the overhead tracker. This intelligent use ofmemory resources means that the approach can run comfort-ably on a standard desktop computer, producing high qualityreconstructions, despite the low cost, low resolution cameras.4. EXPERIMENTAL RESULTSAll experiments in this section were performed a standarddesktop computer; (Windows 32Bit, 3GHz CPU, 4GB RAM).Figure 3(a) shows 3D volumetric reconstruction results, ob-tained from our approach, on a sample frame using betweenone and four cameras. In this example, the voxel resolutionsin the carved volume are set to 1cm3. It should be noted,that if the initial volume extended to the whole tennis court,a voxel resolution of approximately 1000cm3 (one voxel hasa resolution of 10cm×10cm×10cm ) would have to be set inorder for the same amount of memory to be utilised, result-ing in a significant decrease in reconstruction accuracy. Infigure 3, a maximum of four cameras is used, although thereare nine cameras in the network, due to limited camera over-laps, at most the player will appear in four fields of view at agiven moment in time. The graph in figure 4(a) illustrates thetrade-off between reconstruction time and voxel resolution.For example, the process runs at 38 frames per second whenx = 3, i.e. a voxel is 3cm×3cm×3cm, or 27cm3.A difficulty in evaluating a space carving technique thatcovers a large area, such as a tennis court, is the lack of avail-able ground truth data for comparison. In an attempt to pro-vide quantitative evaluation figures, the Mean Squared Error(MSE) of a single frames space carving and its associated sil-houettes are compared in the graph of figure 4(b). In thisgraph, a space carving was automatically generated from a(a) (b)Fig. 4. Quantitative evaluation results; (a) Reconstructiontime versus voxel resolution (where x is the length of eachof a voxel’s faces in cm); (b) Mean-square error (MSE) ver-sus voxel resolution.number of sets of input images. The resultant carving wasthen back-projected into each camera plane, forming a sil-houette of the final volumetric model. For each image, thisback-projected silhouette was compared to a manually seg-mented ground truth silhouette using a Mean Square Errormetric. This qualitative measure was then averaged across allcameras. This process was repeated and graphed for differingvoxel spatial resolutions. As expected, when the voxel spa-tial resolution is decreased, the MSE decreases accordingly.This highlights the importance of using memory efficiently,to acquire the highest possible voxel spatial resolution.5. DISCUSSIONIn this work, we implemented a viable 3D visualisation tech-nique, by means of dynamic voxel carving, and using a lowcost, low resolution, camera network. We demonstrated thisapproach in the context of tennis, however a variety other realworld applications are possible. With the exception of initialcamera synchronisation and calibration the entire process iscompletely autonomous and does not require any manual in-teraction. In future work, we will investigate the possibilityof fitting a 3D human skeleton to the volumetric model andacquiring dynamic motion data associated with an athlete’sbody. The acquisition of such data from athletes on the fieldof play would allow to quantify performance data, such asvelocities, ground reaction forces and torques, amongst otherdata to be obtained for evaluation of sporting performance.6. ACKNOWLEDGEMENTSThis work is supported by Science Foundation Ireland undergrant 07/CE/I1147 and was partially supported by the Euro-pean Commission under contract FP7-247688 3DLife.7. REFERENCES[1] C. O´ Conaire, D. Connaghan, P. Kelly, N. E. OConnor,M. Gaffney, and J. Buckley, “Combining inertial andvisual sensing for human action recognition in tennis,”in ACM ARTEMIS, 2010, p. 51.[2] A. Broadhurst, T.W. Drummond, and R. Cipolla, “Aprobabilistic framework for space carving,” in Intern.Conference on Computer Vision, 2001, vol. 1, p. 388.[3] W. B. Culbertson, T. Malzbender, and G. Slabaugh,“Generalized voxel coloring,” in Intern. Workshop onVision Algorithms: Theory and Practice, 1999, pp. 100–115.[4] K. N. Kutulakos and S. M. Seitz, “A theory of shape byspace carving,” Intern. Journal of Computer Vision, vol.38, pp. 199–218, July 2000.[5] S. M. Seitz, B. Curless, J. Diebel, D. Scharstein, andR. Szeliski, “A comparison and evaluation of multi-view stereo reconstruction algorithms,” in IEEE Con-ference on Computer Vision and Pattern Recognition,2006, vol. 1, pp. 519–528.[6] S. Vedula, S. Baker, S. Seitz, and T. Kanade, “Shapeand motion carving in 6d,” in IEEE Conference on Com-puter Vision and Pattern Recognition, June 2000, p. 592.[7] R. Yang, M. Pollefeys, and G. Welch, “Dealing withtextureless regions and specular highlights-a progressivespace carving scheme using a novel photo-consistencymeasure,” in IEEE Conference on Computer Vision,2003, vol. 2, p. 576.[8] V. Kolmogorov and R. Zabih, “Multi-camera scene re-construction via graph cuts,” in European Conferenceon Computer Vision-Part III, 2002, pp. 82–96.[9] S. Bhatia, L. Sigal, M. Isard, and M. J. Black, “3dhuman limb detection using space carving and multi-view eigen models,” in IEEE Computer Society Confer-ence on Computer Vision and Pattern Recognition Work-shops, 2004, p. 17.[10] C. Nitschke, A. Nakazawa, and H. Takemura, “Real-time space carving using graphics hardware,” IEICE -Trans. Inf. Syst., vol. E90-D, pp. 1175–1184, 2007.[11] G. K.M. Cheung, T. Kanade, J-Y Bouguet, andM. Holler, “A real time system for robust 3d voxel re-construction of human motions,” in Computer Visionand Pattern Recognition, 2000, vol. 2, p. 714.[12] Y-K Yang, J. Lee, S-K Kim, and C-H Kim, “Adap-tive space carving with texture mapping,” in ICCSA (3),2005, pp. 1129–1138.[13] Ciaran O Conaire, Philip Kelly, Chanyul Kim, andNoel O’Connor., “Automatic camera selection for ac-tivity monitoring in a multi-camera system for tennis,”in ACM/IEEE Intern. Conference on Distributed SmartCameras, 2009.

Dynamic voxel carving in tennis based on player localisation using a low cost camera network

O\u27Connor, Noel E.

Irish Universities

Name not available

http://doras.dcu.ie/16322/1/Camera_ready_2011_ICIP.pdf

Dynamic voxel carving in tennis based on player localisation using a low cost camera network

Abstract

Similar works

Full text

Available Versions

DCU Online Research Access Service

Irish Universities

Name not available