Bartoli, P.

Colins, Toby

Fisher, Bob

English

Edinburgh Research Explorer

     Edinburgh Research Explorer                                      Automatic Quasi-Isometric Surface Recovery and Registrationfrom 4D Range DataCitation for published version:Colins, T, Fisher, B & Bartoli, P 2008, 'Automatic Quasi-Isometric Surface Recovery and Registration from4D Range Data'. in Proceedings of the BMVA Symposium on 3D Video - Analysis, Display and Applications.BMVA Press.Link:Link to publication record in Edinburgh Research ExplorerDocument Version:Publisher final version (usually the publisher pdf)Published In:Proceedings of the BMVA Symposium on 3D Video - Analysis, Display and ApplicationsGeneral rightsCopyright for the publications made accessible via the Edinburgh Research Explorer is retained by the author(s)and / or other copyright owners and it is a condition of accessing these publications that users recognise andabide by the legal requirements associated with these rights.Take down policyThe University of Edinburgh has made every reasonable effort to ensure that Edinburgh Research Explorercontent complies with UK legislation. If you believe that the public display of this file breaches copyright pleasecontact openaccess@ed.ac.uk providing details, and we will remove access to the work immediately andinvestigate your claim.Download date: 20. Feb. 2015Automatic Quasi-Isometric Surface Recovery and Registration from 4D RangeDataToby CollinsProf. Robert FisherUniversity of EdinburghDr. Adrien BartoliUniversity of Clermont Ferrand1 IntroductionTo unlock the huge potential in video-rate range image analysis,arguably one the most significant hurdles to overcome is that ofaccurately recovering motion over a given sequence. For a singleobject, this largely amounts to uncovering the isomorphism under-gone by the object’s surface from a set of point sample observationsin each frame. Unfortunately even with 3D data this is a highlychallenging task, since in the absence of salient 3D structure (e.g.the region around a person’s cheek or a section of cloth), motioncannot be resolved over the manifold (much like the aperture af-fect in 2D.) To alleviate this, we must impose assumptions aboutthe object’s deformation and/or appearance sub-space a priori (aconcept well established in 2D image analysis). In 2D these pri-ors must be extremely broad to account for the nonlinear, unknownand largely irrecoverable interaction between the object’s deformedstate and it’s appearance in 2D. In general, the best we can do is touse extremely broad assumptions about smooth 2D motion, whichhas been demonstrated time again to be insufficient to adequatelyconstrain the registration/tracking tasks.By contrast, in our work we have focused on investigating generalclasses of deformation priors in 3D, where the story is very differ-ent. In 3D we can directly reason in terms of true 3D dynamics,and our assumptions on the underlying isomorphism can lead tovery well defined registration problems. So well defined in fact thatwe show here how to go beyond mere registration (i.e. matchingeach frame to a particular template) to the harder problem of au-tomatically recovering a complete, registered 3D model given onlylimited deformed observations. Solutions to this would find con-siderable application in the computer vision and graphics commu-nities, where it is desirable to obtain object models that undergonon-rigid motion (e.g. a flying flag or a deforming body), under-going occlusion (both self and external), never fully observable andpresent in cluttered scenes.In this work our attention is focused on one important class of de-forming objects; those whose surfaces undergo isometric or near-isometric deformations. These are a particular kind of isomorphismwhich relates a surface’s embedding in 3D space (i.e. a Riemannianmanifold) by a transformation preserving distances on the manifold(geodesics). Qausi-isometric transformations describe well the de-formations undergone by a large range of real-world objects, suchas many fabrics, paper, plastics, articulated objects and, to an ex-tent, facial expressions. Robust methods for detecting and regis-tering objects of this class therefore have uses in several fields in-cluding computer graphics, such as texture extraction, texture map-ping, deformation transfer and video augmentation; computer vi-sion, such as unsupervised deformable object learning, tracking,and deformation analysis.2 Method Overview and ContributionFrom a sequence of range observations our overarching goal is toautomatically reconstruct the surfaces of deforming objects appear-ing in the sequence. This task is nontrivial for a number of rea-sons. Firstly, occlusion boundaries in range images, particularlythose generated by stereo, cannot be reliably used for segmenta-tion. Occlusion cues rely on depth discontinuities, which are oftensmoothed by the stereo algorithm, or not present if the occludingobject is in contact with the surface. Without reliable segmenta-tion, any priors on the isomorphism (including isometry or smooth-ness assumptions) are not valid (and thus detrimental) over regionswhich contain occlusion zones. The second problem that of globalnonrigid alignment. Even if given a correct segmentation, matchingpairs of range segments involves locally solving the non-rigid reg-istration task. Without correspondences known a priori, this gen-erally results in motion models with dense Jacobian and Hessianmatrices. Scaling this up to a global alignment readily becomesintractable using numerical optimisation, yet this is highly desir-able since local methods tend to result in global misalignments (asdemonstrated in, for example 2D panorama stitching.)However, we can obtain a foothold to the problem by consider-ing certain deformation-invariant properties over the deforming sur-face. Here we present a global method for nonrigidly aligning sur-face segments located in range data that mutually agree with respectto the assumed isomorphism model. For near-isometry, this is thepreservation of geodesic lengths on the manifold. We essentiallyperform surface model completion by embedded mosaicing, wherea composite is formed in the surfaces intrinsic coordinate space.This has the desirable property that deformed segments are now re-latable by far simpler transformations. For quasi-developable sur-faces (i.e. isometric surfaces with very low Gaussian curvature suchas cloth), isometry reduces to near-Euclidean transformations.The key stages to our approach for automatic deformable surfacereconstruction are shown in figure 1, and a brief overview of eachstage is described in the following sections.Figure 1: Deformable model recovery and registration frameworkSince video rate range sequences contain a vast amount of redun-dant surface information, we would prefer to sample those frameswhich convey the most new surface information with which to buildthe model. Initially this set is selected using simple strategies (e.g.every nth frame or random selection) from the range sequence, withfurther frame being added and included in the model based on (i)agreeing with the current model and (ii) revealing unseen sectionsof the surface.2.1 Robust Feature CorrespondencePoint correspondences are often key components in registrationtasks largely for guiding the registration towards global optima.Typically, feature matchers in 2D (e.g. SIFT) and 3D (e.g. spinimages) will misalign, so some form of correspondence annealingFigure 2: Deformable surface reconstruction from multiple range observations. TL: Regions matched and extracted from several rangeimages (some of which are shown in TR. BL the resulting reconstruction comparing (left) the true surface texture, (middle) rigidly alignedpatches and (right) the nonrigidly aligned and stitched model. BL: An example distortion map for a particular segment.Figure 3: Robust point correspondence between range pairs using spectral clustering. From left to right: Correspondences matched by SIFT;Robust correspondence and clustering using our method; Spectral embedding of matches (green indicates outliers); Another example.is often employed. We instead establish correspondence by appeal-ing to the isometric assumption; that is mutual point distances onthe manifold (geodesics) should be preserved. We establish a mea-sure of isometric conformity between all pairs of candidate corre-spondences , and the optimal solution is the subset set of matcheswith maximal mutual agreement. In our method we find an effi-cient inexact solution to this using spectral methods to find stronglyconnected clusters in a compatibility graph whose nodes representmatches and edges weighted by their mutual match scores. Sinceocclusion zones (undetected or otherwise) break geodesic agree-ments, multiple clusters may exist in the graph which are eachbounded by an occlusion zone. Thus, we use k-way spectral cluster-ing to jointly recover the correct matches and segmentation (Figure3.) Currently, we extract features based only on intensity, since 3Dfeatures on smooth developable surfaces are almost entirely am-biguous.2.2 Quasi-Developable Surface SegmentationGiven a set of segmented high-quality feature correspondences, weproceed to extract the surrounding region agreeing with our as-sumption of developabliy. Although zero Gaussian curvature char-acterises perfectly developable surfaces, its computation is oftentoo unstable to derrive a segmentation. Furthermore, for real sur-faces low Gaussian curvature is usually present. Instead, fromour matched features we grow a region over the range image suchthat the distortion (angular and stretch) induced by flattening it isbounded by some tolerance. Once grown, we further refine theregion using other segmentation cues (i.e. strong intensity/depthgradients and colour histograms), combined proabablistically andsolved using graph cuts.2.3 Global Nonrigid Segment AlignmentGiven multiple surface segments, from multiple range images, webring them into nonrigid alignment by minimising two error cri-teria related to (i) mutual feature distances between matched seg-ments and (ii) mutual feature distances over each segment. Thefirst enforces the alignment of matched features whilst the secondpreserves the system’s rigidity. Since (ii) is very nonlinear (quadricin position), we settle for a slightly weaker interpretation which en-forces conformality rather than rigidity over a segment (i.e. anglepreservation.) This is quadratic in position and results in a full-rankleast squares system that can be solved in closed form using sparselinear least squares. The second stage propagates the transforma-tions from the features through each segment. Since our underly-ing assumption is of near-rigidity on the 2D plane, we use an as-rigid-as-possible transformation similar to that proposed recentlyby [Schaefer et al. 2006] used in interactive shape manipulation.2.4 Low-Distortion Surface Texture RecoveryOur final stage of recovering the surface model is to generate a ren-dering of the surface’s texture. Given the match correspondences,shading artifacts can be removed using standard techniques. Togenerate the render, blending techniques used in image mosaicingare not suitable, due to residual ghosting from small misalignments.Instead we are driven by the goal of a low-distortion rendering. Inour method we greedily select and stitch those regions which werewere transformed onto the plane with the least distortion. Given aninitial patch (i.e. the one least distorted), we treat the addition of asecond patch as a binary labeling problem, combining a distortionpenalty with a seam cost, and solve this incrementally using graphcuts.2.5 RegistrationIn order to fit the model to the rest of the data, we adopt an en-ergy minimisation process which penalises states that diagree indata evidence (i.e. 3D distance to the point cloud and feature corre-spondence), and a quadratic isometric within-plane bending model[Bergou et al. 2006]. For all other frames, we perform the samestrategy but initialise the model to its deformed state in the pre-vious frame. Global drift is avoided by using hard feature con-straints between the model and scan as described in section 2.1. Thenewly registered instances can then be further incorporated into ourmodel, along with new neighbouring regions agree with respect tothe model’s isomorphism prior.3 ResultsWe have tested our approach on several sequences capturing de-forming paper and fabrics. Figures 2 and 3 show examples of somegood results attained on a seqeunce of a deforming magazine cover.In the near future we aim to further validate our methods for othersequences. We also aim to extend our work by better modellingthe relationship between mesh distortion and nonrigid deformationconstraints on the 2D plane, and will be investigating our frame-work for recovering other classes of deforming surfaces.ReferencesBERGOU, M., WARDETZKY, M., HARMON, D., ZORIN, D., ANDGRINSPUN, E. 2006. A quadratic bending model for inextensi-ble surfaces. In SGP ’06: Proceedings of the fourth Eurograph-ics symposium on Geometry processing, Eurographics Associa-tion, Aire-la-Ville, Switzerland, Switzerland, 227–230.SCHAEFER, S., MCPHAIL, T., AND WARREN, J. 2006. Imagedeformation using moving least squares. In International Con-ference on Computer Graphics and Interactive Techniques.

Automatic Quasi-Isometric Surface Recovery and Registration from 4D Range Data

Abstract

Similar works

Full text

Available Versions

Edinburgh Research Explorer