ISSN: 1051-4651 Print ISBN: 978-1-4244-7542-1International audienceFinding point correspondences which are consistent with a geometric constraint is one of the cornerstones of many computer vision problems. This is a difficult task because of spurious measurements leading to ambiguously matched points and because of uncertainty in point location. In this article we address these problems and propose a new robust algorithm that explicitly takes account of location uncertainty. We propose applications to Sift matching and 3D data fusion

Sur, Frédéric

INRIA a CCSD electronic archive server

HAL Id: inria-00474083https://hal.inria.fr/inria-00474083Submitted on 20 Sep 2010HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestinée au dépôt et à la diffusion de documentsscientifiques de niveau recherche, publiés ou non,émanant des établissements d’enseignement et derecherche français ou étrangers, des laboratoirespublics ou privés.Robust matching in an uncertain worldFrédéric SurTo cite this version:Frédéric Sur. Robust matching in an uncertain world. 20th International Conference onPattern Recognition - ICPR 2010, IAPR, Aug 2010, Istanbul, Turkey. pp.2350 - 2353,￿10.1109/ICPR.2010.575￿. ￿inria-00474083￿Robust matching in an uncertain worldFrédéric SURÉcole des Mines de Nancy (INPL) & LORIA (Magrit team, INRIA)Campus Scientifique BP 239 - 54506 Vandœuvre-lès-Nancy cedex - Francefrederic.sur@loria.frAbstractFinding point correspondences which are consistentwith a geometric constraint is one of the cornerstonesof many computer vision problems. This is a difficulttask because of spurious measurements leading to am-biguously matched points and because of uncertainty inpoint location. In this article we address these prob-lems and propose a new robust algorithm that explicitlytakes account of location uncertainty. We propose ap-plications to SIFT matching and 3D data fusion.1. Motivation and related workPoint of interest (POI) matching is one of the veryfirst steps of many computer vision tasks. The aim is tofind point correspondences between several views (2Dfor a classic camera or possibly 3D for a depth-camera)of the same physical object or scene. 2D or 3D pointmatching is a prior step to registration, where a trans-formation between views is estimated from the match-ing step. A popular way to tackle the matching problemis to proceed in two separated steps: 1) find out a list ofputative correspondences, based on some features asso-ciated with POI (e.g. 3D shape descriptors or SIFT [7]in 2D), then 2) extract from this list the largest possi-ble subset of correspondences that are consistent with acommon transformation.Point matching is an intrinsically difficult problem.It is indeed impossible to match POI at step 1) withoutintroducing false correspondences (also called outliers,as opposed to inliers), i.e. correspondences betweenpoints that do not actually correspond to the same phys-ical point. Moreover, POI location has limited accuracy.Even with a careful subpixel interpolation, the accuracyof image POI is limited to 0.2 / 0.5 pixel.Robust algorithms (in the sense that they have to dealwith outliers, and possibly to limited accuracy) are thusneeded for step 2). The most popular choice is cer-tainly RANSAC [5] and numerous methods derived fromit. RANSAC is an iterative procedure where candidatetransformations are tested over the set of putative corre-spondences. Incorporating point location uncertainty ina robust matching algorithm is the main topic of severalworks: from 3-D registration [12] to point matching un-der projective transformation [2, 10, 14]. Uncertaintyis somehow incorporated in MLESAC [16], where in-liers location is assumed to be spoilt by an isotropicGaussian perturbation. In [15], MLESAC is adaptedto incorporate the uncertainty on the candidate trans-formations. In [14] we have incorporated the uncer-tainty of the fundamental matrix in the parameterlessstate-of-the-art RANSAC scheme of [9]. Following ourwork, [13] proposes to speed-up RANSAC by limitingthe search for correspondences in the mapped error re-gions. All of these papers assume that POI location un-certainty is isotropic. Characterizing the uncertainty ofPOI location is a quite recent problem in computer vi-sion. A recent paper [17] estimates the location uncer-tainty of the popular SIFT features [7] and demonstratesthe soundness of the approach.Our contribution is a general RANSAC-like robust al-gorithm, which explicitly takes into account the uncer-tainty of POI. It is inspired by the a contrario modelfrom [9]. The proposed algorithm is general in the sensethat it is well suited to 2D or 3D correspondences (con-trary to existing methods), with broad potential appli-cations. Still contrary to most existing methods, we donot a priori assume that point uncertainty is isotropic.In particular, the proposed algorithm can be applied forSIFT matching with the uncertainty derived from [17].2. Gaussian error and Mahalanobis metricWe describe here the point matching problem what-ever the dimensionality d and the transformation group(rotations, homographies, etc). We assume that a listof putative correspondences S made of N couples ofd-dimensional points (xi, yi)1≤i≤N has been given bysome algorithm exploiting feature similarity (step 1)).The aim of a robust matching algorithm is to separatethis list between inliers (for which there exists a map-ping A such that yi ' Axi) and outliers.The xi’s (resp. yi’s) are modeled as random vari-ables following a Gaussian distribution with mean xiand covariance matrix Σxi (resp. yi and Σyi ). We notexi ∼ N (xi,Σxi). Let us remark that xi and yi are un-known, we only observe the uncertain POI xi and yi.In the presented framework, A is a parametric map-ping, which needs at least p correspondences for its esti-mation. For example, in the case of planar data (d = 2),estimating a homography requires 4 correspondences,and 5 for 3D data. In RANSAC, at each iteration A isestimated from exactly p correspondences. Since pointlocation is uncertain, estimation of A is also uncertain.The aim of this section is to define a distance be-tween yi and Axi taking account of the uncertainty. Wewill use the following classic theorem which is the key-stone of most papers handling uncertainty:Proposition 1 (propagation property) Let v be a ran-dom vector in Rd, v ∼ N (v,Σ), and f : Rd → Rd bean affine mapping such as f(v) = f(v) + A(v − v).Then f(v) ∼ N (f(v), AΣAT ).When f is not linear, a first order Taylor series approxi-mate gives an estimate of the covariance by replacingAby the Jacobian J(v) of f at v.2.1. Uncertainty propagationWe estimate here the uncertainty ofA·x, whenA is arandom d×d Gaussian matrix with meanA and covari-ance ΣA (it is a d2 × d2 matrix), while x ∼ N (x,Σx).Uncertainty of A. A is estimated from p uncor-related correspondences (x1, y1), . . . (xp, yp). A =(ai,j)1≤i,j≤d is seen here as the column vector(a1,1, . . . a1,d, . . . . . . ad,1, ad,2, . . . ad,d)T.By Prop. 1, ΣA can thus be written as:ΣA = J ·∆(xi, yi)1≤i≤p · JT (1)where ∆ is the 2dp × 2dp block-diagonal matrix madeof the Σxi and Σyi , and J is the d2 × 2dp Jacobianmatrix of the mapping: (x1, y1, . . . xp, yp) 7→ A withrespect to the d components of the p points xi and yi.Let us remark that the computation of J has tobe adapted to the actual algorithm estimating A fromthe p correspondences. With Direct Linear Trans-form [6], A is the solution of a Singular Value Decom-position (SVD), and J then comes from the Jacobian ofthe SVD [11]. In [14] we have estimated by a closed-form formula the uncertainty of the fundamental matrixobtained by the linear 8 point algorithm and SVD. Inthis paper, we estimate J by a finite-difference scheme.Uncertainty of A(x), which is a random variableas A and x. If A is linear, estimating the Jacobian ofthe bilinear mapping (A, x) 7→ A · x is straightforward.It is the d × (d2 + d) matrix: J̃ =(Dx A)whereDx is the d×d2 matrix diag(xT , xT . . . xT ). Assumingthat A and x are uncorrelated (which holds if x is notused in estimating A), the covariance of A · x can bederived (using Prop. 1) as:ΣA·x = Dx · ΣA ·DTx +A · Σx ·AT . (2)If A is not linear, the Jacobian matrix is explicitly com-puted to replace J̃ , and then permits to propagate thecovariance in Eq. (2). Calculus is here easily tractable.2.2. Estimating the Mahalanobis distanceWe now define a distance between y ∼ N (y,Σy)and Ax ∼ N (Ax,ΣAx), when x and y are correspond-ing through A. Assuming y = Ax and independencebetween the random variables, then y−Ax has mean 0and covariance matrix Σy + ΣAx. Assuming the ran-dom variables are Gaussian, a popular way to measurethe similarity between y and Ax is to use the so-called(squared) Mahalanobis distance [8]:dM (y,Ax) = (y−Ax)T (Σy + ΣAx)−1 (y−Ax) (3)which follows a χ2 law with d degrees of freedom (dof).The independence assumption holds if (x, y) doesnot intervene in the estimation of A. To the best of ourknowledge, such a use of the Mahalanobis distance hasbeen first studied in [12]. The value of this distance isnever used in robust matching: for example in [2, 10,13] it is just used to define a search region once a fixedconfidence level is set a priori.The metric is symmetrized with:dsM (x, y) = dM (y,Ax) + dM (x,A−1y) (4)which follows a χ2 law with 2d d.o.f. under indepen-dence assumption.How to actually compute dsM? Eq. (2) wouldgive ΣA−1y , provided ΣA−1 is known. This latter co-variance can be derived from ΣA. For example, if Ais a rotation matrix, then A−1 = AT and ΣA−1 isjust a reordering of the coefficients of ΣA. If A isa (linear) affine transformation, then A−1 is the in-verse of A and ΣA−1 can be estimated from A viathe non-linear propagation property. In this case, onejust needs the Jacobian matrix of the mapping from Ato A−1. Since d(A−1) = −A−1dAA−1: (differenti-ate AA−1 = I), one has: ∂(A−1)∂Ei,j= −A−1Ei,jA−1where Ei,j is the canonical base matrix with entriesequal to 0 except in position (i, j) which is 1. If Ais a homography, A−1 is still the inverse of A inhomogeneous coordinates, but the normalisation step(here ||A||2 = 1) must be taken into account in the prop-agation.3. A contrario RANSAC for uncertain pointsOne of our contribution is to propose a robust algo-rithm for uncertain point matching via a method basedon a so-called a contrario model. Books [3] and [4]give a survey of a contrario models in many computervision problems. The idea behind a contrario modelsis that independent, structureless random features canproduce structured groups only with a very small proba-bility. However, though it is rarely pointed out, it is alsothe underlying idea of every method using the χ2 lawto set a threshold over the Mahalanobis distance basedon the 5% significance: found correspondences are un-likely under independence assumption.Let us be more specific. As in every RANSAC-like al-gorithm, a candidate transformation A is first estimatedfrom a minimal set s made of p correspondences. Afitness measure for a subset S of S containing s is:dmax(A,S, s) = max(xi,yi)∈SdsM (xi, yi) (5)or dsum(A,S, s) =∑(xi,yi)∈SdsM (xi, yi). (6)We now define the null hypothesisH0.Definition 1 H0: (xi, yi)i∈S\s are independent Gaus-sian random variables, xi ∼ N (xi,Σxi), yi ∼N (yi,Σyi), also independent of the random map-ping A ∼ N (A,ΣA) such that ∀i, yi = Axi.Let δ be some positive number. Thanks to indepen-dence assumption among points in S\s and between A(estimated over s) and points of S\s, one computes, if kdenotes the cardinality of S:Pr (dmax(A,S, s) ≤ δ|H0) =(fχ22d(δ))k−p(7)and Pr (dsum(A,S, s) ≤ δ|H0) = fχ22(k−p)d(δ) (8)where fχ2α is the cumulative distribution function (c.d.f)of the χ2 law with α d.o.f. Indeed, the c.d.f. of the maxof (k − p) i.i.d. variables is the common c.d.f. to thepower (k − p), and the sum of k − p i.i.d. variablesfollowing a χ2 law with 2d d.o.f. follows a χ2 lawwith 2(k−p)d d.o.f. We will use dsum in the sequel; dmaxgives similar results.In the a contrario methodology, one does not directlydeal with the probabilities but rather with the so-calledNumber of False Alarms (NFA).Definition 2 A set S of correspondences is ε-meaningful if there exists a mapping A such that:NFA(A,S) := (N − p)(kp)(Nk)fχ22(k−p)d(δ) ≤ ε(9)where δ is the observed value of dsum(A,S, s).This quantity automatically balances accuracy andnumber of points in S (ε-meaningful sets with larger kmust have smaller δ), while the standard method con-sisting in thresholding eq. 5 or 6 via a fixed significancerate over-favours large k. Def 2 is motivated byProposition 2 The expected number of ε-meaningfulset underH0 is lower than ε. (see [3, 4])Remark. Now, from Eq. 2, 3, and 4, one remarks thatthe larger the uncertainty of A (or A−1), the smallerthe Mahalanobis distance. In order to prevent to fallinto degenerate cases (for example homographies withpoints in s aligned), one cannot just minimize the NFA.Our aim is actually twofold: minimizing both the NFAand the uncertainty of A and A−1. We thus decide tobound from above the uncertainty of the tested trans-formation A when searching for a most meaningful set.This was also remarked in [13].Algorithm. Iteratively sample p correspondencesfrom S and derive the corresponding A. If the largesteigenvalue of ΣA or ΣA−1 is above some threshold(10.0 in Sec. 4), then go to the next iteration. Other-wise build the subset S ⊂ S with the lowest NFA byremarking (as in [9]) that when k is fixed, the subset ofcardinality k with the lowest NFA is made of the k cor-respondences among S with the smallest dsM distances.In the end, return the subset with the lowest NFA.4. Proof-of-concept experimentsSIFT matching under homography. SIFT uncer-tainty comes from [17]. Fig. 1 and 2 (top) showstwo views with POI putative correspondences superim-posed. Feature matching (step 1) in Sec. 1) is voluntar-ily set to produce many outliers. Results are shown withthe covariances Σy + ΣAx and Σx + ΣA−1y (90% el-lipsoid, yellow) superimposed to POI. One can see thattaking into account the Mahalanobis distance automati-cally adapts the inlier/outlier threshold: it is gentler forpoints with higher uncertainty (e.g. because they aredistant from the 4 points estimating A - in red). For ex-ample, bottom-left POI in Fig. 2 would not have beenmatched with classic RANSAC. In Fig. 2 the algorithmis run twice: once the set with the lowest NFA has beenfound, the algorithm is run on the whole set of corre-spondences minus the first group. Both groups (consis-tent with a homography) correspond to aligned planes.(The cathedral and the building frontage are actuallyaligned, which explains the matches in the 2nd group).3D point cloud merging. We propose here an exper-iment about 3D data fusion based on synthetic data.The aim is to identify the 3D homography between twoclouds of matched points, which contain outliers. Weplan to test our algorithm on 3D reconstructions of ascene obtained by Structure From Motion algorithms orby depth-cameras. Propagating the uncertainty of SIFTPOI or of the depth-camera measurements to the 3Dpoints could indeed provide valuable information forpartially overlapping reconstructions [1].Experimental setting. 1,000 points are uniformlydrawn in a 100 × 100 × 100 block, and separated be-tween inliers (I) and outliers (O). Inliers xi are trans-Figure 1. Top: 44 SIFT correspondences. Many out-liers can be seen. Bottom: 10 inliers are retrieved withour algorithm. Please zoom in on the pdf file.formed by a 3D homography into the yi, and outliersare associated with uniformly drawn points into the 3Darea limited by the inliers. Each point is given a randomcovariance which may have standard deviation up to 6in a direction, and its position is changed with a stan-dard deviation of 3 (compare to the size of the block: theproblem is quite challenging). The table gives statisticsfor our algorithm (average over 10 runs) in 6 situations.# I # O # retrieved # O among retrieved1,000 0 154.8 0900 100 122.7 1.4700 300 89.3 1.6500 500 61.8 4.5300 700 45.5 6.9100 900 27.5 17.5While the algorithm misses a large amount of inliers,the number of outliers among the retrieved correspon-dences is excellent, even with large rate of outliers inthe dataset. Interestingly, if one stops the algorithm assoon as a 1-meaningful group is retrieved (this is soundbecause of prop. 2), the retrieved group has about 20-30correspondences and is returned on average after lessthan 4 iterations (up to 50% outlier rate). This makes usexpect possible densification strategies as in [13].Acknowledgements: thanks are due to Gilles Simon andNicolas Noury for providing the images, and to Marie-Odile Berger for numerous valuable discussions.References[1] G. Boström, J. Gonalves, and V. Sequeira. Controlled3D data fusion using error-bounds. JPRS 2008.[2] S. Brandt. On the probabilistic epipolar geometry. IVC2006.[3] F. Cao, J. Lisani, J.-M. Morel, P. Musé, and F. Sur. Atheory of shape identification. Springer 2008.[4] A. Desolneux, L. Moisan, and J.-M. Morel. FromGestalt theory to image analysis. Springer 2008.Figure 2. Top: 160 initial SIFT matches. Middle: 1stgroup (41 matches). Bottom: 2nd group (20 matches).[5] M. Fischler and R. Bolles. Random sample consensus:A paradigm for model fitting. Comm. of the ACM 1981.[6] R. Hartley and A. Zisserman. Multiple View Geometryin Computer Vision. Cambridge University Press 2000.[7] D. Lowe. Distinctive image features from scale-invariant keypoints. IJCV 2004.[8] P. Mahalanobis. On the generalised distance in statistics.Proc. of the Nat. Inst. of Sciences of India 1936.[9] L. Moisan and B. Stival. A probabilistic criterion to de-tect rigid point matches between two images and esti-mate the fundamental matrix. IJCV 2004.[10] B. Ochoa and S. Belongie. Covariance propagation forguided matching. SMVP 2006.[11] T. Papadopoulo and M. Lourakis. Estimating the Jaco-bian of the Singular Value Decomposition: theory andapplications. In ECCV 2000.[12] X. Pennec and J.-P. Thirion. A framework for uncer-tainty and validation of 3D registration methods basedon points and frames. IJCV 1997.[13] R. Raguram, J.-M. Frahm, and M. Pollefeys. Exploitinguncertainty in random sample consensus. ICCV 2009.[14] F. Sur, N. Noury, and M.-O. Berger. Computing the un-certainty of the 8 point algorithm for fundamental matrixestimation. BMVC 2008.[15] B. Tordoff and R. Cipolla. Uncertain RanSaC. MVA2005.[16] P. Torr and A. Zisserman. MLESAC: A new robust es-timator with application to estimating image geometry.CVIU 2000.[17] B. Zeisl, P. Georgel, F. Schweiger, E. Steinbach, andN. Navab. Estimation of location uncertainty for scaleinvariant feature points. BMVC 2009.

Robust matching in an uncertain world

http://ieeexplore.ieee.org/xpls/abs_all.jsp?arnumber=5597003

Robust matching in an uncertain world

Abstract

Similar works

Full text

Available Versions

INRIA a CCSD electronic archive server