Aanæs, Henrik

Guilbert, Nicolas

Larsen, Rasmus

A new approach for formulating prior knowledge in structure form motion is presented, where the structure is viewed as a 3D stochastic variable, hereby priors are more naturally expressed. It is demonstrated that this formulation is efficient for regularizing structure reconstruction via prior knowledge. Specifically algorithms for imposing priors in the proposed formulation are presented

Online Research Database In Technology

INTEGRATING PRIOR KNOWLEDGE AND STRUCTURE FROM MOTIONN. Guilbert, H. Aanæs and R. LarsenIMMTechnical University of Denmark ( DTU )2800 Lyngby, DenmarkE-mail: proj61@imm.dtu.dkABSTRACTA new approach for formulating prior knowledge in struc-ture from motion is presented, where the structure is viewedas a 3D stochastic variable, hereby priors are more natu-rally expressed. It is demonstrated that this formulation isefficient for regularizing structure reconstruction via priorknowledge. Specifically algorithms for imposing priors inthe proposed formulation are presented.1. INTRODUCTIONStructure from motion is a field of research which allowsthe reconstruction of the 3D structure and motion of a rigidbody from a sequence of two or more images where 2D fea-tures of interest have been detected and correlated throughthe frames, see e.g. [3]. It is an essential tool in artificialintelligence and image understanding, applicable in robotvehicle navigation, content-based searches in video data orarchitectural visualization.However, it is an inherent property of the approach, thatthe accuracy of 3D reconstruction is underpinned by thenoise in the images, be it due to quantization or shortcom-ings of the feature extraction algorithm. Hence, enhancingthe 3D structure via prior knowledge can improve the resultsconsiderably. For instance, knowing that a 3D object pri-marily consists of planes in right angles to each other, likeeg. buildings, can greatly enhance its reconstruction. Onemight also envisage integrating a more sophisticated modelsalong the line of [1, 2], whereby more complex priors canbe expressed and applied.Previously, Baker et al. [6] and Torr et al. [5] have ad-dressed the problem of incorporating prior knowledge intostructure from motion. These approaches identify layersin the images and constrain the reconstruction accordingly.These methods use backprojection onto the image(s) to de-termine whether regularization with a given prior is proba-ble. Consequently, the noise is expressed in the 2D imagedomain, and it is implied that the camera matrices are deter-mined perfectly.In this paper, we address the problem of combining pri-ors with structure from motion by formulating the estimatedreconstruction as 3D a stochastic variable. This approachhas the advantage of being intuitively accessible, since the3D structure is computed explicitly. The explicite recon-struction also allows handling of the uncertainty of the cam-era motion, since our approach can incorporate that the noiseon a given 2D feature potentially affects all of the 3D fea-tures. Finally, integration of more complex priors than planes(eg. splines or deformable templates) is natural.We establish the mean of the 3D structure by standardstructure from motion methods, see e.g. [3]. The disper-sion of the resulting structure is then obtained by approx-imating the 2D to 3D transformation by a linear function,whereby the relations between noise on the 2D features andthe 3D structure becomes apparent. In order to validate ourapproach, we then impose the prior of planar surfaces intostructure from motion, applying a clustering technique.The organisation of this paper is as follows: section 2describes the stochastic structure modeling approach, sec-tion 3 describes the algorithm for fitting a plane to a setof points given heteroscedastic and anisotropic noise. Sec-tion 4 concerns the clustering of smaller planes into the finalplane estimates.2. STOCHASTIC STRUCTURE MODELA mentioned, we view the estimated 3D structure as a stochas-tic variable, derived from the the tracked 2D feature pointsin two images. We assume a calibrated pinhole camera:xijsij= AiRiti0 1 Xj1= AiPiXj1(1)where the 2D feature xij, is the projection of the 3D featureXj, in image i. Image i is described by its calibration matrixAi, its rotation Riand translation tirelative to the worldcoordinate system.From the tracked 2D features, and this observation model(1), it is well known that the 3D structure, Xjand the cam-era motion, Pi, can be calculated via the epipolar geometry,see e.g. [3].As such, assuming a well posed problem, the 3D struc-ture:X = [X1; : : : ; Xn]can thus be seen as a function,F , of the tracked 2D features:x = [x11; : : : ; x1n; x21; : : : ; x2n]i.e.F : x! XSo viewing X and x as stochastic variables with as-sumed Gaussian noise such that:Xj2 N(^Xj;j) xij2 N(x^ij;2Dij)we would like to estimateX given x. Since we do not haveF in closed form, we approach this problem by linearizingF , by means of numerical gradients. Hence:X 2 F(x)  F(^x) +XkÆFÆxkN(0;2Dij)) (2)V ar(X) XkÆFÆxk2V ar(xk) (3)E(X) = F(x) (4)denoting the composing features of x by xk– i.e. k is validconfigurations of ij – and assuming that the noise on x isindependent. Hence from (3) the Gaussian noise structureofX can be derived.As noted, this derived noise structure of X, is based onthe approximation of (independent) Gaussian noise, and oflinearizing F . These assumptions seam reasonable, in thatthe approach i capable of capturing, how variations in theimage data, x, affects the estimated structure, X. Theseaffects on the estimated structure,X, are all what is reason-able to expect, since the noise structure on the 2D featuresis rarely known and as such all distributions – e.g. Gaussian– are approximations.Our assumption of a calibrated camera –Aiknown – isby no means necessary, as seen in [3], except if the imposedpriors are non invariant to protective transformations, whichthey will seldom be.3. ESTIMATING THE PLAN (PRIOR)In order to test if a set of given 3D features lie on the sameplane, and if so enforce that plane upon the the structure, weneed to estimate the most likely plane. In this case it is non–trivial, since the 3D features have different or heterosced-stisk anisotropic noise. Implying that the Mahalanobis dis-tance measure or norm for each 3D feature is different.Hence given m 3D features fX1; : : : ; Xmg, and cor-responding Gaussian variance structures f1; : : : ;mg wewant to find the plane that minimizes the distance between3D feature Xjand the plane, in the norm induced by  1j.Let a plane be denoted by its normal vector, , and its offsetfrom origo, 0, then any point, X , on the plane satisfies:TX + 0= 0The most likely plane is estimated by iteratively approxi-mating the noise structures by heteroscedastic isotropic noise,until convergence is achieved. In this isotropic case, themost like plane is given by:0=  Pmj=1jXjPmj=1j = min0mXj=1jjj(Xj+ 0)T 0jj22(5)where jdenotes the isotropic variance of j. Given aplane, the jare given as the ratio between the length ofthe minimum distance between 3D feature and plane, ~rj, inthe induced norm relative to the minimum distance in the2-norm, rj, i.e.j=~rTj 1j~rjrTjrj(6)In short, the algorithm is, where q denotes iterations:1. Initialize q = 0 ; 8j 0j= detj.2. Estimate Plane q ; q0with isotropic noise, for (5).3. Update Isotropic Noiseqjvia (6)4. If not Stop, q = q + 1, goto 2. The stop criteria ismaxj(qj  q 1j) < tolerance:It is seen, that the jare updated such that the givenplane has the right likelihood, with the original heterosced-stic anisotropic noise. So if 8j qj= q 1jthen the op-timal plane with the isotropic noise is also optimal in theanisotropic case.4. IMPOSONG THE PRIORAs mentioned, we validate our approach by imposing theprior of planar surfaces onto our structure. This is done byfirst triangulating the estimated 3D structure, whereby a setof three–point planes are constructed. A recomended meth-ods for triangulation is [4] by Morris and Kande.As such, this triangulation does not reguaralize the 3Dstructure, but it serves as a initialization for an algorithmwhere neighboring planes (e.g triangles) which are likely tobe the same plane are merged. A statistical test for copla-narity is described below.4.1. Test for CoplanarityGiven a set of 3D features, as described above, located on aplane, and this plane is estimated optimally as described inSection 3, then the residuals are realizations of the respec-tive noise, N(0;j), as estimated in Section 2. Hence a testof two sets of 3D featuresX1;X2being located in the same?Fig. 1.plane is to compare the residuals for the plane fitted to oneof the sets and the plane fitted to the two sets combined.More formally, the entities that should be compared arethe residuals, ~rjbetween a 3D feature and the plane, normedwith the respective variance, j. Since, if the assumptionof coplanarity holds, these residuals should be elements ina N(0; 1) distribution. Hence a F-test can be used, or for-mally if the assumption thatX1andX2are part of the sameplane holds, then:mnPni=1~r1Tj 1j~r1jPmi=1~r12Tj 1j~r12j2 F(n;m) (7)where ~r1 are the n residuals from X1and ~r12 are the mresiduals fromX1[X2.4.2. Clustering AlgorithmThe algorithm for combining neighboring planes is a greedyclustering algorithm in that at any given time, the two neigh-boring planes with the must likelihood of being the sameplane are combined, if this likelihood is above a given thresh-old. Hence the triangles achieved by triangularization, areclustered into planes.When these clusters have been calculated, the derivedplanes are enforced upon the 3D features, and the 3D fea-tures moved onto the plane, whereby the prior of planarstructures is imposed. See ??5. RESULTS6. CONCLUSION7. ACKNOWLEDGMENTSStreet respect to Allan A˚sbjerg Nielsen for enlightened dis-cussions on plane fitting with heteroscedastisc anisotropicnoise.8. REFERENCES[1] A. Blake and M. Isard. Active Contours. Springer–Verlag,London, UK., 1998. 352 pp.[2] T. F. Cootes, C. J. Taylor, D. H. Cooper, and J. Graham. Ac-tive shape models – their training and application. ComputerVision, Graphics and Image Processing, 61(1):38–59, January1995.[3] R. Hartley and A. Zisserman. Multiple View Geometry. Cam-bridge University Press, The Edinburgh Building, CambridgeCB2 2RU, UK, 2000.[4] D. Morris and T. Kanade. Image-consistent surface triangu-lation. In IEEE Conf. Computer Vision and Pattern Recogni-tion’2000, pages 332–338, 2000.[5] A. R. Dick P. H. S. Torr and R. Cipolla. Layer extractionwith a bayesian model of shape. In European Conf, ComputerVision’00, pages 273–289, June 2000.[6] R. Szeliski S. Baker and P. Anandan. A layered approach tostereo reconstruction. In IEEE Conf. Computer Vision andPattern Recognition’98, pages 434–441, 1998.

Integrating prior knowledge and structure from motion

http://orbit.dtu.dk/ws/files/2950155/imm206.pdf

Integrating prior knowledge and structure from motion

Abstract

Similar works

Full text

Available Versions

Online Research Database In Technology