This paper introduces a technique for extracting structure and motion using directionally selective matches between linear features. A world-centered coordinate system is used to make these computations without the intermediate calculation of depth. In order to constrain the possible structure and motion configurations, we assume that the three-dimensional direction of gravity relative to each image frame is known. The direction of gravity, along with the directionally selective linear feature matches, form a set of quadratic equations which can be used to determine structure and motion

Gardner, Warren F.

Lawton, Daryl T.

Scholarly Materials And Research @ Georgia Tech

Shape and Motion from Linear FeaturesWarren F. Gardner and Daryl T. LawtonCollege of ComputingGeorgia Institute of TechnologyAtlanta, GA 30332-0280AbstractThis paper introduces a technique for ex-tracting structure and motion using direction-ally selective matches between linear features.A world-centered coordinate system is usedto make these computations without the in-termediate calculation of depth. In orderto constrain the possible structure and mo-tion congurations, we assume that the three-dimensional direction of gravity relative toeach image frame is known. The direction ofgravity, along with the directionally selectivelinear feature matches, form a set of quadraticequations which can be used to determinestructure and motion.1 IntroductionThe extraction of environmental structure and motionfrom a sequence of two-dimensional images is a com-mon problem in computer vision. Typically solutions tothis problem are expressed in camera-centered coordi-nate systems where environmental geometry is speciedby the depth along an image feature's ray of projection.Unfortunately, parameters computed from this camera-centered representation are dependent upon the depth toenvironmental features. This leads to erroneous resultsfor objects located far from the camera.The recently introduced factorization method [Tomasiand Kanade, 1990; Tomasi and Kanade, 1992; Boult andBrown, 1992] has attempted to overcome the disadvan-tages associated with a camera-centered representation.This method uses a world-centered coordinate system,along with an orthogonal projection assumption, in orderto compute shape and motion without the intermediatecalculation of depth. A matrix of image measurementsis constructed by making point correspondences betweenimage frames. The matrix is then factored into a shapematrix and a motion matrix using Singular Value De-composition.One problem with the factorization method is that itrelies upon accurate point correspondences between im-This research is supported by the Advanced ResearchProjects Agency of the Department of Defense and is mon-itored by the U. S. Army Topographic Engineering Centerunder contract No. DACA76-92-C-0016age frames. This paper introduces a method of extract-ing shape and motion from directionally selective lin-ear feature correspondences. This line-based algorithmis capable of reconstructing shape and motion withoutcomputing depth as an intermediate step. In addition tothe orthogonality assumption, we assume that the three-dimensional direction of gravity is known relative to eachimage in a motion sequence.The algorithm begins by searching for the orientationof one of the lines in the environment. This is a onedimensional search over 180o, constrained by the projec-tion of the line on one of the image planes. Each candi-date line orientation, along with the position of gravity,forms a set of quadratic equations which constrain allthe other lines, as well as the rotation between imageframes. An error measure is computed from the derivedline orientations and used to evaluate each shape andmotion conguration. Once the line orientations andparameters of rotation have been derived, the relativepositions of the lines can also be computed from simplelinear equations.The remainder of this section introduces the notationused throughout this paper. Section 2 shows how toderive line orientation and camera rotation from a se-quence of two-dimensional images. Section 3 presents aset of linear equations which can be used to solve for therelative line positions. The algorithms presented in thepaper are applied to synthetic data and the results arepresented in Section 4. Finally, concluding remarks aregiven in Section 5.1.1 NotationThe notation used throughout this paper is shown inFigure 1. An image frame at time f is delineated byunit vectors if , jf , and kf . A three-dimensional envi-ronmental line is represented by a unit vector ds speci-fying the line direction, and a point on the line ps. Line(ds; ps) is projected orthographically onto image framef . The direction of the projected line is represented byits unit normal ~nfs. ~pfs refers to the projection of ps.The direction of gravity will be referred to as gf . Thetwo-dimensional parameters ~nfs and ~pfs as well as thethree-dimensional parameter gf are all expressed in thecoordinate system of image frame f . All other parame-ters are specied relative to the world coordinate system.When ~nfs is specied in the world coordinate system itFigure 2: Normals are determined by intersecting a planewith a circular coneSince the line normals ~nf1 were formed by orthographicprojection, they must be perpendicular to the line d1.Therefore, one constraint is that the vectors nf1 mustlie within the plane perpendicular to d1. An additionalconstraint is provided by the gravity vector gf . The an-gle between ~nf1 and gf must be the same as the anglebetween nf1 and the direction of gravity in the worldcoordinate system (gw). These two constraints can beused to solve for nf1. Figure 2 shows the geometry ofthese two constraints. Each normal (nf1) is determinedby intersecting a plane with a circular cone. The planeis dened by d1. The cone is constructed by rotating avector about the direction of gravity at the appropriateangle. Since the origin of the cone lies within the plane,the intersection of the plane with the cone results in twolines. There are only two possible solutions since thenormals are known to be unit vectors.The constraints described above will now be examinedin more detail. As stated earlier, the direction of gravitygf relative to the line normals ~nf1 is known. This resultsin the following relationshipnf1  gw = ~nf1  gf (1)where gw is the direction of gravity in the world coor-dinate system. Letting gw = (0; 1; 0) we can simplifyEquation 1nf1y =  ~nf1  gf (2)In addition to the angle constraint we know that nf1lies within the plane dened by d1. This constraint isexpressed asnf1  d1 = 0 (3)Finally, we know that the magnitude of each normal vec-tor (nf1) equals oneknf1k = 1 (4)Equations 2, 3, and 4 can be combined into a singlequadratic equation, resulting in two feasible solutionsfor each normal vector.2.2 Additional Line NormalsThe next step in the extraction of line orientation androtation is to solve for the position within the world co-ordinate system of the rest of the line normals. This isFigure 3: Normals are determined by intersecting twocircular conesaccomplished by using the candidate line normals. Theidea is essentially the same as in the previous section.Two constraints can be formulated from the given geom-etry. The rst constraint is given by the gravity vectorgf , and is identical to the constraint presented in theprevious section. The angle between ~nfs and gf mustbe the same as the angle between nfs and the directionof gravity in the world coordinate system. The secondconstraint is that the angle between an image normalvector ~nfs and the candidate image normal vector ~nf1must be the same as the angle between the associatedworld coordinate normal vectors nfs and nf1. These twoconstraints can be used to solve for all the additionalnormal vectors nfs. The constraints are shown geomet-rically in Figure 3. The solution for a normal vector nfsis essentially the result of intersecting two circular cones.One cone is the result of rotating a vector about the di-rection of gravity. The other cone results from rotatinga vector about the candidate normal vector nf1. Theintersection of two circular cones which share the sameorigin is two lines. Once again, the normals are knownto be unit vectors, resulting in two solutions.The following equations result from the above analy-sis. The constraint resulting from the gravity vector gfis identical to the one presented in Section 2. Therefore,from Equation 2 we can writenfsy =  ~nfs  gf (5)The second constraint relates the line normals nfs to thecandidate line normals nf1 as followsnfs  nf1 = ~nfs  ~nf1 (6)Finally, we know that the magnitude of each normal vec-tor (nfs) equals oneknfsk = 1 (7)Equations 5, 6, and 7 can be combined into a singlequadratic equation, resulting in two feasible solutionsfor each normal vector.2.3 Parameter EstimationOnce the normal vectors (nfs) have been derived, theprocess of estimating the line orientations and rotationalparameters is trivial. The line orientations (ds) are easilyestimated from their associated normals (nfs) using thefollowing equationds  nfs = 0 (8)ds can be estimated with a minimumof two non-collinearnormal vectors. When more vectors are available, dscan be solved for using a linear least-squares technique.The rotational parameters are also easily obtained fromthe normal vectors nfs. Three linear equations can beformulated for the three rotational parameters if , jf ,and kfif  nfs = ~nfsxjf  nfs = ~nfsykf  nfs = 0There are also additional constraints available. One ofthese constraints is that the vectors must be orthonormalif = jf  kfjf = kf  ifkf = if  jfkifk = kjfk = kkfk = 1Additional constraints can be derived from the relation-ship between the rotational vectors and gravity as wasdone in Sections 2.1 and 2.2. These constraints areif  gw = gfxjf  gw = gfykf  gw = gfzOf course, all of the equations presented above are notindependent, and all are not necessary. Currently we usethe following subset of equations. Initially kf is deter-mined using a least squares formulation ofkf  nfs = 0 (9)The technique presented in Section 2.1 is then used tosolve for if with the following equationsif  kf = 0 (10)if  gw = gfx (11)Finally if and kf are used to solve for jfjf = kf  if (12)Equation 8 is used to solve for the line orientationsds. Equations 9, 10, 11, and 12 are used to solve forthe rotational parameters if , jf , and kf . The followingsection shows how to use these derived parameters tosolve for the relative positions of the line segments, thuscompleting the spatial reconstruction.3 Line PositionThe nal step in the line segment reconstruction is tosolve for the line segment positions relative to the worldFigure 4: The rst and last frames from a 20 imagesequencecoordinate system. Initial assumptions about the posi-tion of the image frames relative to the world coordinatesystem are made, allowing a simple linear solution tothe problem. The position of each line is representedby a point ps which is chosen arbitrarily. The worldcoordinate system will be positioned at the center of im-age frame 1. The points ~p1s are then chosen arbitrarily~p1s = (xs; ys). We assume that all the image planes in-tersect along line d1. This means that the position ofeach image plane is given by p1 + fds where f is aparametric scale factor.Each line position ps = (xs; ys; zs) consists of one un-known zs. The solution for zs is trivial. Each point psis constrained to lie within the planes perpendicular tonfs. These planes are positioned by choosing some ar-bitrary point on the projection of each line, and thendetermining the position of that point within the worldcoordinate system. Let q be the point in world coordi-natesq = p1 + [if jf ]  (~pfs   ~pf1) (13)The equation of the plane is then written asnfsx(xs   qx   fdsx)+nfsy (ys   qy   fdsy) +nfsz (zs   qz   fdsz ) = 0 (14)The two unknowns in this equation are zs and f . fcan be removed from the equation, and a least squaressolution can be found for zs.4 ResultsThe algorithm presented in this paper was implementedand tested on several sequences of synthetic data. Therst and last frames from a 20 image sequence are shownin Figure 4. Figure 5 shows 10 frames from the sequence(every other frame is displayed). This data was pro-duced by random rotations and translations. The ro-tational parameters if and jf associated with this se-quence of motion are shown in Figures 6 and 7. Thecorrect rotational values are displayed as solid lines, andthe derived values are displayed as dotted lines. All er-rors are the result of perspective projection. Notice thatFigure 5: 10 image frames from a 20 image sequenceY-componentX-componentZ-componentFigure 6: The components of if for a 20 frame sequence.The correct values are shown with solid lines, and thederived values are shown with dotted lines.the Y-component of if is errorless. This is because thiscomponent is derived from the relationship between theimage frames and the gravity vector (gf ) as shown inEquation 11. Thus the Y-component is unaected bythe perspective projection errors.The derived line orientations and parameters of rota-tion were then used to reconstruct the line positions asdiscussed in Section 3. A top view of the original datais shown in Figure 8. The reconstructed data is shownin Figure 9. Once again the errors are the result of per-spective projection.5 ConclusionThe technique presented in this paper is an early attemptat constructing linear feature based depth-independentmotion algorithms. The work has only been tested onsynthetic data, and it is not clear what eect perspectiveZ-componentX-componentY-componentFigure 7: The components of jf for a 20 frame sequence.The correct values are shown with solid lines, and thederived values are shown with dotted lines.Figure 8: Top view of the house dataprojection and other forms of noise will have. However,since the formulation involves linear least squares esti-mation, it appears that it will be robust. The ability todeal with occlusion is also straight-forward in this over-constrained system. Occluded line normals (nfs) are nullvectors and therefore have no eect on the least squaressolution. Notice that the rst frame shown in Figure 4contains occluded lines.One drawback of this method is that the three-dimensional direction of gravity is required. This mea-surement can be provided by a gravity sensor, but wewould like to relax this restriction. One way to removethe gravity vector from the algorithm is to replace the di-Figure 9: Top view of the reconstructed houserection of gravity with another consistent direction. Forexample, for an object that consistently moves in onedirection (such as a vehicle), the gravity vector can bereplaced by a vector specifying this direction (the for-ward vehicle direction).There are several areas for future work: Test this algorithm on noisy data and if necessarydevelop a more robust formulation that will workwell in the presence of errors, including the errorsintroduced from perspective projection. Test the algorithm on real image sequences. Integrate this rotation based method with thetranslation based method discussed in [Lawton,1982]. In this case the gravity vector is replaced bya direction of translation vector. The integration ofthese two methods will probably be accomplishedthrough temporal ltering using the Kalman Filter.References[Boult and Brown, 1992] Terrance E. Boult andLisa Gottesfeld Brown. Motion segmentation usingsingular value decomposition. In Proceedings of theDARPA Image Understanding Workshop, pages 495{506, 1992. San Diego, CA.[Lawton, 1982] Daryl T. Lawton. Processing transla-tional motion sequences. Computer Vision, Graphics,and Image Processing, 22:116{144, 1982.[Tomasi and Kanade, 1990] Carlo Tomasi and TakeoKanade. Shape and motionwithout depth. In Proceed-ings of the DARPA Image Understanding Workshop,pages 258{270, 1990. Pittsburgh, PA.[Tomasi and Kanade, 1992] Carlo Tomasi and TakeoKanade. The factorization method for the recoveryof shape and motion from image streams. In Proceed-ings of the DARPA Image Understanding Workshop,pages 459{472, 1992. San Diego, CA.

Shape and Motion from Linear Features

https://smartech.gatech.edu/bitstream/1853/3613/1/93-06.pdf

Shape and Motion from Linear Features

Abstract

Similar works

Full text

Available Versions

Scholarly Materials And Research @ Georgia Tech