International audienceThis paper presents a new surface reconstruction method that extends previous carving methods for non-Lambertian objects by integrating the smoothness and image information of different aspects. We introduce a robust multiview photo-consistency function and a single-view visualconsistency function. The former considers a general specularity that may introduce both reflectance models and robust statistics, while the latter is based on pixel homogeneity and continuity in individual images. The new consistent shape, with both multi-view and single-view, extends previous photo hull and visual hull, and is proven to be included in both photo hull and visual hull. This approximate shape is then refined by a graph-cut optimization method. Sample reconstructions from real sequences for both global and local optimizations are demonstrated

Paris, Sylvain

Quan, Long

Zeng, Gang

INRIA a CCSD electronic archive server

HAL Id: inria-00510170https://hal.inria.fr/inria-00510170Submitted on 13 Oct 2010HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Robust Carving for Non-Lambertian ObjectsGang Zeng, Sylvain Paris, Long QuanTo cite this version:Gang Zeng, Sylvain Paris, Long Quan. Robust Carving for Non-Lambertian Objects. Proceedings ofInternational Conference on Pattern Recognition (ICPR), 2004, Cambridge, United Kingdom. ￿inria-00510170￿Robust Carving for Non-Lambertian ObjectsGang ZENG1 Sylvain PARIS2 Long QUAN11 Dep. of Computer Science, HKUST, Clear Water Bay, Kowloon, Hong Kong{zenggang, quan}@cs.ust.hk2 ARTIS/ GRAVIR-IMAG, INRIA Rhône-Alpes, 38334 Saint Ismier, Francesylvain.paris@imag.frAbstractThis paper presents a new surface reconstruction methodthat extends previous carving methods for non-Lambertianobjects by integrating the smoothness and image infor-mation of different aspects. We introduce a robust multi-view photo-consistency function and a single-view visual-consistency function. The former considers a general spec-ularity that may introduce both reflectance models and ro-bust statistics, while the latter is based on pixel homogene-ity and continuity in individual images. The new consistentshape, with both multi-view and single-view, extends previ-ous photo hull and visual hull, and is proven to be includedin both photo hull and visual hull. This approximate shapeis then refined by a graph-cut optimization method. Sam-ple reconstructions from real sequences for both global andlocal optimizations are demonstrated.1. IntroductionThree-dimensional surface reconstruction from imagesis a fundamental but challenging problem and has been in-tensively studied. Most existing techniques are based on theLambertian assumption: the intensity values of reflected ra-diance of a surface point are equal in all directions fol-lowing a cosine rule, producing view-independent effects.However, this rule is clearly insufficient for most commonmaterials that have more complex behaviors like highlight,reflection, transparency or subsurface scattering. Handlingall these effects in a unified way still appears unachievable,as some of them duplicate the characteristics of other ob-jects. This paper focusses on opaque and non-reflective ob-jects and proposes a general approach without any specificconstraint. The range of materials we consider is thereforebroad: plastic, unpolished metal, general cloth, skin, etc. Wefirst describe some existing work without being exhaustive.Methods based on non-Lambertian assumptionsA practi-cal way of handling non-Lambertian objects is to removehighlights to transform them into Lambertian ones. Lin etal. [13] remove highlights from images with a color his-togram difference. Magda et al. [15] propose two originalmethods that exploit specularity instead of simply remov-ing its. However, for their method, a specific light and cam-era setup is needed. Carceroni and Kutulakos [6] proposea method of recovering a 3D shape using dynamic surfel.Georghiades [9] proposes a surface reconstruction methodbased on the Torrance-Sparrow reflectance Model. Bhat andNayar [2] have studied the relationship among camera con-figurations, surface properties, and specular intensity varia-tions. Their work is based on a specific model, and givesmajor qualitative ideas: the matching tolerance has to behigher with a wider baseline and/or with shinier surfaces.Methods based on different image informationSilhouetteis employed to construct an approximate shape called vi-sual hull [12,16]. More precise approach [22] is proposed toevaluate local curvatures along occluding contours. Photo-consistency is widely used for volumetric reconstructionmethods [11, 18, 19] by comparing colors under the Lam-bertian assumption. However, photo-consistency is not ro-bust, it fails as soon as strong non-Lambertian objects arepresent. Recently, several methods [1,3,5,7] have been pro-posed to improve the above carving methods based on aprobabilistic framework. Texture correlation considers tex-ture similarities and can be used in level set methods [8] orgraph-cut methods [4, 10, 17, 23]. Cross-correlation, partic-ularly its zero-mean normalized version, ZNCC, is more ro-bust as it is invariant to local linear transformation of light-ing. The computation of ZNCC is better if the surface orien-tation is taken into account since it is operating on a largerwindow.Overview of our approachWe propose a novel approachto integrate various techniques into two steps to combinetheir advantages while avoiding their drawbacks. We firstuse multi-view and single-view consistency criteria to re-construct an overall shape of the object without any pre-vious information but lacks precision. We then use ZNCCto refine this shape into a finely detailed surface. This sec-ond step relies on the orientation information of the shapein the first step. We also introduce two novel graph-cut tech-niques that make the refinement more precise and robust.2. Robust multi-view photo-consistencyThe first step toward reconstructing non-Lambertian ob-jects is to define a robust carving criterion from multipleviews using reflectance models. We choose the Phong re-flectance model for our first attempt. Although it is not aphysical model, it is simple and works correctly under thespecificity of our setup. In the case of surface reconstructionfrom multiple images, we combine the Lambertian modeland the Phong model to describe both view-independentand view-dependent effects:I(v) = α +∑βi(ri · v)σwherev is the unit vector in the viewing direction andriis the unit vector in the mirror reflection (maximum reflec-tion) direction of each light source, withα andβi control-ling the quantity of the Lambertian and non-Lambertian ef-fects.σ indicates the shininess of the material.Like traditional photo-consistency, the criterion of multi-view consistency is defined as the mean deviation betweenthe colors predicted by the reflectance model and the col-ors actually seen from the images:MV C = minα,βi,σ{√1k∑(Ij − I(vj))2}wherek is the number of visible cameras, withIj being theprojection on thejth visible image andvj being the corre-sponding viewing direction.In our algorithm,σ is selected from a set of user-definedparameters. For example, we need two parameters for headreconstructions to indicate the shininess of hair and skinseparately (e.g. [21] provides some typical values for skin).We also notice that in most cases, one light source is enoughto model the reflection because there is only one significanthighlight. Thus, we approximater by the viewing directionof the camera in which the point appears the brightest.Finally, we simply ignore the worst matches to allow asignificant but not major deviation from the Phong model.We handle image noise: we estimate the criterion througha process that starts with the above classical form and re-fines it by iteratively singling out samples whose distanceto the current model is higher than the others. We will con-sider more accurate models in our future work.MV Cr = minα,β,σ{√1k − r∑wj(Ij − I(vj))2}with wj = 0, 1 and∑wj = k − r3. Single-view visual-consistencyEven if multi-view photo-consistency is made robust asabove, one of the major problems is that the continuity ofthe object has never been taken into consideration. One ofthe possibilities for enforcing the continuity is to go backto individual images, as these sophisticated non-Lambertianview-dependent effects are stable in individual images. Ourgoal is then to enforce that, in every image, the silhouette ofthe object corresponds to some image discontinuities. Thisis what we callsingle-view consistency[24].We combine the multi-view and single-view consistencyas follows. When a voxel is found to be inconsistent andcarved by multi-view, we check immediately whether thecurrent shape is single-view consistent or not. Since wehave an over-estimated 3D shape, all projected 2D sil-houettes are also over-estimations of the object boundariesand thus they have to move inward to ensure that crossedcolor-homogeneous regions become purely background re-gions. Therefore, the parts outside the “visual hull” of theshrunken silhouettes are carved, and the new 3D estima-tion is enforced to inside this hull. The algorithm terminateswhen all surface voxels are multi-view consistent, and wethen have a consistent shape, calledphoto-visual hull.4. Surface refinement by texture correlationThe photo-visual hull gives a tight estimation of the ob-ject, which also provides visibility and geometry informa-tion for the surface points. To obtain a more detailed sur-face, we let the current surface evolve in the surroundingnarrow band and aim at finding the surface for which thetexture correlation gives the best score. This score has beenadapted to account for the non-Lambertian specificity of ourtechnique. We choose the Zero-mean Normalized Cross-Correlation (ZNCC) as it is invariant to a local linear trans-formation of lighting. Thus, it gives an accurate measure-ment of the texture correlation even if view-dependent ef-fects are present. Based on the visibility and geometry in-formation of the photo-visual hull, we select the two mostfront-facing visible cameras to compute this correlationscore in order to limit the perspective distortion.However, we cannot straightforwardly select the bestmatched points because they may introduce irregularity.This caveat is solved by the introduction of a smoothingconstraint through an optimization process which removesthe irregularity according to the neighborhood.4.1. Global optimizationWe choose the general graph-cut optimization intro-duced in [20]. In the case of a single object with surroundingviewpoints, we naturally employ a cylindrical parametriza-tion (h, θ, r) of the object and aim at finding an optimal sur-facer(h, θ) minimizing the following functional:∫∫N(c(h, θ, r(h, θ)) + a |rh(h, θ)|+ br|rθ(h, θ)|)r dθ dh(1)wherec(·) is the consistency term of the surface point, witha andb controlling the smoothing constraint.(a) (b) (c) (d) (e)(a) (c) (d) (e)(b)Figure 1. (a) Photo-visual hull. (b) Global optimization. (c-d) Local optimization. (e) Comparison with inputs.This technique introduces a 3D embedded graph whosemain property is the equivalence between itsminimal cutand the optimal surface of the functional equation (1). Itspurely geometric formulation makes its adaptation to oursetup straightforward.4.2. Local optimizationAnother possibility is to enforce the optimization to runin a local context since sub-parts of the object may be inde-pendent and therefore can be reconstructed separately. Forinstance, in the case of head reconstructions, the hair can bereconstructed without considering the eyes. Thus, we runlocal optimizations several times in different positions in-stead of a global optimization. Although these local opti-mizations are run independently, adjacent optimizations arelinked since their neighborhoods largely overlap.Based on the tight estimation,re(h, θ), we per-form the following local optimization for every surfacepoint (hi, θi, re(hi, θi)). We first define its local con-text N to be of size4h × 4θ × 4r and centered atthis point. Then, the graph-cut optimization [20] is per-formed within this local context by minimizing the func-tional (1). Finally, the valuer(hi, θi) is assigned accordingto the center of this optimal surface patch.5. Experimental results and discussionsWe usually acquire about 35 images roughly in all di-rections with a hand-held camera by circumnavigating theobject. The lighting and the background are arbitrarily un-known. The geometry of the sequences is automaticallycomputed and self-calibrated from a standard uncalibratedapproach [14]. It is also important to note that these kindsof sequences are typically difficult for traditional carvingmethods: the image appearance changes quite a lot fromone view to another; skin and hair are widely known to behighly non-Lambertian due to numerous complex effects.Our implementation runs successfully on many examples.Here we show two typical head modeling examples, onewith a more textured face and the other with less. The re-construction results are shown in Figures 1 and 2 includ-ing the tight estimation and two optimized shapes. The roleof each step of the algorithm is clearly put into evidence.The tight estimation (fig. 1,2-a) gives a very robust but yetnot sufficiently accurate location of the object. The single-view consistency has segmented out the background whilethe multi-view consistency recovers some concavities. Thishas been achieved fully automatically without any prior in-formation of the background. Then, detailed surface geom-etry (fig. 1,2-b,c,d) is carved out in the second step usinggraph-cut optimizations. We see particularly the very accu-rate recovery of concavities around the hair and eye areas.Global optimization vs local optimizationComparing theglobal optimization (fig. 1,2-b) with the local optimization(fig. 1,2-c), we see the smoothness of the former and thesharpness of the latter. The global one yields more reason-able results on smooth regions, including the forehead andthe cheeks, but fails to get details on sharp regions, includ-ing the nose, the eyes, the mouth and the boundary of thehair. The local one produces detailed geometry with localcontinuity, resulting in satisfactory surfaces.6. ConclusionsWe have proposed a novel approach for reconstructingnon-Lambertian objects from an arbitrary set of calibratedimages. To achieve our goal, we first generalize the photo-consistency criterion into a robust multi-view consistency,which combines reflectance models and robust statistics.(a) (b) (c) (d) (e)(b) (c) (d) (e)(a)Figure 2. (a) Photo-visual hull. (b) Global optimization. (c-d) Local optimization. (e) Comparison with inputs.Then, we introduce a new single-view aspect, which intrin-sically overcomes view-dependent caveats. This is mainlyrelated to robust computation of silhouettes and appears asa complementary tool to multi-view consistency. The abovetwo steps result in a tighter shape estimation which includesboth photometric and contour information. This shape is fi-nally refined thanks to the usage of graph-cut optimizationsdriven by texture correlation. This approach has been vali-dated on various reconstructions, and is shown to be espe-cially efficient on human heads, which are widely known tobe highly non-Lambertian.AcknowledgementsThis project is supported by the Hong Kong RGC grantHKUST 6188/02E.References[1] M. Agrawal and L. Davis. A probabilistic framework for sur-face reconstruction from multiple images.CVPR 01.[2] D. Bhat and S. Nayar. Stereo in the presence of specular re-flection. ICCV 95.[3] R. Bhoitka, D. Fleet and K. Kutulakos. A probabilistic theoryof occupancy and emptiness.ECCV 02.[4] Y. Boykov and V. Kolmogorov. Computing Geodesics andMinimal Surfaces via Graph Cuts.ICCV 03.[5] A. Broadhurst, T. Drummond and R. Cipolla. A ProbabilisticFramework for Space Carving.ICCV 01.[6] R. Carceroni and K. Kutulakos. Multi-view scene capture bysurfel sampling.ICCV 01.[7] J. DeBonnet and P. Viola. Poxels: Probabilistic voxelized vol-ume reconstruction.ICCV 99.[8] O. Faugeras and R. Keriven. Complete dense stereovision us-ing level set methods.ECCV 98.[9] A. Georghiades. Incorporating the Torrance and SparrowModel of Reflectance in Uncalibrated Photometric Stereo.ICCV 03.[10] V. Kolmogorov and R. Zabih. Multi-camera scene recon-struction via graph cuts.ECCV 02.[11] K. Kutulakos and S. Seitz. A theory of shape by space carv-ing. ICCV 99.[12] A. Laurentini. The visual hull concept for silhouette-basedimage understanding.TPAMI 94.[13] S. Lin, Y. Li, S. Kang, X. Tong, and H. Shum. Diffuse-specular separation and depth recovery from image sequences.ECCV 02.[14] M. Lhuillier and L. Quan. Quasi-dense reconstruction fromimage sequence.ECCV 02.[15] S. Magda, T. Zickler, D. Kriegman, and P. Belhumeur. Be-yond lambert: Reconstructing surfaces with arbitrary BRDFs.ICCV 01.[16] W. Matusik, C. Buehler, R. Raskar, S. Gortler and L. McMil-lan. Image-based visual hulls.SIGGRAPH 00.[17] S. Roy. Stereo without epipolar lines : A maximum-flow for-mulation. IJCV 99.[18] S. Seitz and C. Dyer. Photorealistic scene reconstruction byvoxel coloring.IJCV 99.[19] G. Slabaugh, B. Culbertson, T. Malzbender, and R. Schafer.A survey of methods for volumetric scene reconstruction fromphotographs.VolumeGraphics, 2001.[20] S. Paris, F. Sillion and L. Quan. A Surface ReconstructionMethod Using Global Graph Cut Optimization.ACCV 04, ex-tended version submitted to IJCV.[21] S. Paris, F. Sillion and L. Quan. Lightweight Face Relight-ing. Pacific Graphics, 2003.[22] R. Vaillant and O. Faugeras. Using extremal boundaries for3-D object modeling.TPAMI 92.[23] G. Zeng, S. Paris, L. Quan and M. Lhuillier. Surface Recon-struction by Propagating 3D Stereo Data in Multiple 2D Im-ages.ECCV 04.[24] G. Zeng and L. Quan. Silhouette extraction from multipleimages of an unknown background.ACCV 04.

Robust Carving for Non-Lambertian Objects

https://hal.inria.fr/inria-00510170/file/icpr04.pdf

Robust Carving for Non-Lambertian Objects

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server