Nowadays, many applications, ranging from medicine to video games, require accurate 3D geometry and motion of human actors

Laga, H.

Nakajima, M.

Park, J.

English

Research Repository

Multi-view shape reconstruction on CAVE SystemJunyoung Park∗Computer Science Dept.Tokyo Inst.of TechnologyHamid Laga†Global Edge InstituteTokyo Inst.of Technology.Masayuki Nakajima‡Computer Science Dept.Tokyo Inst.of TechnologyFigure 1: Our CAVE System acquires a multi-view image set of a target under 9 different illumination. From a image set, we reconstruct asurface shape of the target.1 IntroductionNowadays, many applications, ranging from medicine to videogames, require accurate 3D geometry and motion of human actors.Existing commercial motion capture systems record only the statusof the markers that are attached to the human body. These pro-vide a sparse representation of the true shape and therefore post-processing steps are necessary to recover the full-body movement.In this paper, we propose a new shape acquisition technique for 3Dsurface reconstruction. The capture system uses eight synchronizedvideo cameras and 3 light projectors to record a subject. The lightprojectors are used in a CAVE-like setup to provide a illuminationpattern. The proposed 3D reconstruction algorithm recovers accu-rate 3D geometry by integrating the depth and normal maps into asingle linear system which can be solved efficiently. Depth mapsare calculated using a space-time multiview stereo matching on theacquired set of images after projecting on them some illuminationpatterns. Surface normal map is estimated using an example-basedphotometric stereo technique[Hertzmann and Seitz 2005] that usesillumination patterns. We show also that an efficient GPU imple-mentation of the stereo matching and photometric stereo algorithmsresults in a speedup of order 20 compared to CPU implementation.2 Hardware Setup and Illumination DesignHardware Setup We use CAVE system as our measurementstage. Our CAVE system consists of 3 rear projection screens, and3 DLP Projectors. Each of the projector projects light to each rearprojection screen. Every screen is fully syncronized and control-lable during camera capture process. To avoid the temporal changesof projected light color, we removed rotary color filter from projec-tors in advance. Removing rotary color filter also gives us morestrong light intensity. We captured a shape of subject using fullysyncronized 8 firewire cameras(PointGrey Flea2, which can cap-ture 30Hz with 1024 × 768 resolution). 8 cameras are placed onaround the CAVE screen. Figure[2] shows our setup.There are some benefits to use CAVE system as a measurementstage. At first, we can get good diffused surface light from everyprojection screen. It reduce the occlusion problem from directionallight. And the large surface light guarantees good signal to noise∗e-mail: junyoung@img.cs.titech.ac.jp†e-mail: hamid@img.cs.titech.ac.jp‡e-mail: nakajima@img.cs.titech.ac.jpFigure 2: Left : Our CAVE setup with a projector which projectslight to front screen. The left and the right screens have same setupas the front screen. Right : Our CAVE System as a measurementstage. Eight cameras are placed on around the CAVE screen.ratio than a bright point light source even the dynamic range ofcamera is narrow. And because we use a projector as a light source,every surface light can be fullly controlable in temporally and spa-tially. Therefore we can make fully controlable light patterns in themeasurement stage.Prior to the measurement, We calibrated our cameras in a geometricway. External and internal camera parameters are calculated usingthe multi-camera calibration technique. In our hardware setup, onepixel in the camera project approximately 1.5mm around the sub-ject area.Time-multiplexed illumination We employed a binary lightpattern to measure the shape of target. We divided each screen in 3vertical areas, and illuminated one after another. Because we syn-chronized our screens and cameras, we know the light condition oneach captured image. Figure[3] shows our illumination patterns.Figure 3: Our Illumination Patterns on 3 CAVE Screen. Eachscreen illuminated in 3 different area by turns, and 3 screen illumi-nated one by one.1693 Single View Surface Reconstruction3.1 Depth EstimationWe ready a set of images on our time-multiplexed illumination pat-tern. The images are rectified from camera calibration. To esti-mate depth map from a image set, we extended a stereo matchingtechnique of [Goesele et al. 2006] to a spacetime stereo matchingtechnique which considers not only the spatial parameters but alsothe temporal parameters. Every depth map value has a confidenceparameter which represents the accuracy of the depth value.3.2 Normal EstimationWe use example-based photometric stereo approach[Hertzmannand Seitz 2005] to estimate surface normals. We use white dif-fuse sphere as a reference object and calculate the normal directionon every camera pixel. We capture the sphere on full illumination,and find a center of circle on image plane using hough transform.From the circle center rays on every camera, we can calculate a mid-position of sphere center using least squares fitting, then reprojectit to each image plane. The reprojected spehre center has a normalwhich is an opposite to camera ray direction. When we calculatea normal of a point in sphere image, we first find a grazing anglepoint which intersects between silhouette of sphere and a line whichpasses through the sphere center and target pixel. Then we can cal-culate the sphere normal on the grazing angle point, and interpolatebetween the grazing angle normal and sphere center normal usingslerp(Figure.4).Glazing angleOptical AxisCenter of ProjectionImage PlanePrincipal PointCenter of ProjectionImage PlanePrincipal PointOptical AxisFigure 4: Normal direction of sphere on perspective view.We capture a reference object on our illumination pattern to builda lookup table between the normal and a normalized observationvector(OV). The OV is a set of intensities observed at a pixel overthe illumination pattern. To estimate a surface normal of a targetobject, we ready a normalized OV under same illumination pattern,search for the closest vector on the look up table, then retrieve therelated normal.3.3 Surface ReconstructionIn the previous section, we uses a multi-view stereo mathcing andphotometric stereo to estimante the depth map and the normal mapof the target object respectively. In general, multi-view stereomatching suffers to estimate a corresponding point on smooth sur-face. On the other hand, photometric stereo suffers to estimate anormal map on noisy textured surface. To reconstruct a surface oftarget, we have to consider to recover these weak points. We willtranscribe the value of depth map and normal map on pixel (i, j) asD(i, j) and N(i, j) respectively.We define a vector which starts from a 3D point on a pixel(i,j) andto other 3D point on the neighbor pixel(i+1, j). the vector is per-pendicular to the normal of the pixel(i,j).(r⃗i,jdi,j − r⃗i+1,jdi+1,j) · N⃗i,j (1)Similarly, in the vertical direction:(r⃗i,jdi,j − r⃗i,j+1di,j+1) · N⃗i,j (2)If we propagate these two constraints for all of the pixels, we get atwice of constraints than the number of unknown parameters di,j .In addition, we define more constraints from depth map informa-tion. We select a high accuracy depth points on depth map, andinsert the depth value as constraints.di,j = Di,j (3)Then we solve the overdetermined linear system using QR decom-position. The solution is a least squars of these constraints. Be-cause our linear system designed to use depth information directly,this linear equation is not an ill posed problem, so we can directlycompose the multi-view surface with a littel effort.4 Conclusions4.1 GPU ImplementationWe implemented the depth / normal estimation methods on GPUusing CUDA. Both of them are well adapted on parallel comput-ing. Comparing with non-parallel calculation, the depth estimationmethods are speeds up approximately 20 times faster, and normalestimation methods are 50 times faster than non-parallel calcula-tion. We implemented the surface reconstruction method on notGPU but CPU because it uses largely sparse matrix calculation. Werun our algorithm on a quad core PC(Intel Core i7 930, 2.8GHz)and a nVidia geForce GTX260 Graphics Card as a GPU. It takesaround 5 minute to reconstrct one set of images.4.2 Result and future workComparing to the state of the art surface reconstrction method of[Vlasic et al. 2009], Our surface reconstruction method is moresimple and reliable because we does not handle the surface withany assumption.In the future work, we will evaluate our method comparing withother surface reconstruction methods.ReferencesGOESELE, M., CURLESS, B., AND SEITZ, S. M. 2006. Multi-view stereo revisited. In CVPR ’06: Proceedings of the 2006IEEE Computer Society Conference on Computer Vision andPattern Recognition, IEEE Computer Society, Washington, DC,USA, 2402–2409.HERTZMANN, A., AND SEITZ, S. M. 2005. Example-based photo-metric stereo: Shape reconstruction with general, varying brdfs.IEEE Trans. Pattern Anal. Mach. Intell. 27, 8, 1254–1264.VLASIC, D., PEERS, P., BARAN, I., DEBEVEC, P., POPOVIĆ, J.,RUSINKIEWICZ, S., AND MATUSIK, W. 2009. Dynamic shapecapture using multi-view photometric stereo. In SIGGRAPHAsia ’09: ACM SIGGRAPH Asia 2009 papers, ACM, New York,NY, USA, 1–11.170

Multi-view shape reconstruction in CAVE system

https://researchrepository.murdoch.edu.au/id/eprint/33600/1/multiview.pdf

Multi-view shape reconstruction in CAVE system

Abstract

Similar works

Full text

Available Versions

Research Repository