Camera calibration is a critical problem in applications such as augmented reality and image based model reconstruction. When constructing a 3D model of an object from an uncalibrated video sequence, large amounts of frames and self occlusions of parts of the object are common and difficult problems. In this paper we present a fast and robust algorithm that uses a divide and conquer strategy to split the video sequence into sub-sequences containing only the most relevant frames. Then a robust stratified linear based algorithm is able to calibrate each of the subsequences to a metric structure and finally the subsequences are merged together and a final non-linearoptimization refines the solution. Examples of real datareconstructions are presented.Postprint (author’s final draft

Bagherzadeh, N

Susín Sánchez, Antonio

Sáinz, M.

English

UPCommons. Portal del coneixement obert de la UPC

ABSTRACTCamera calibration is a critical problem in applicationssuch as augmented reality and image based modelreconstruction. When constructing a 3D model of an objectfrom an uncalibrated video sequence, large amounts offrames and self occlusions of parts of the object arecommon and difficult problems. In this paper we present afast and robust algorithm that uses a divide and conquerstrategy to split the video sequence into sub-sequencescontaining only the most relevant frames. Then a robuststratified linear based algorithm is able to calibrate eachof the subsequences to a metric structure and finally thesubsequences are merged together and a final non-linearoptimization refines the solution. Examples of real datareconstructions are presented.INTRODUCTIONIn recent years Image Based Modeling and Rendering(IBMR) techniques have demonstrated the advantages ofusing real image data to greatly improve the renderingquality in virtual environments. New rendering algorithmshave been presented that reach a photorealistic quality atinteractive speeds when rendering 3D models by usingimages of real objects and some additional shapeinformation (i.e. a geometric proxy). While these methodshave emphasized the rendering speed and quality, theygenerally require extensive preprocessing in order to obtainaccurately calibrated images and geometric proxies of thetarget objects. Moreover, most of these algorithms requireuser interaction for the camera calibration and imageregistration part or need the use of expensive equipmentsuch as calibrated gantries and 3D scanners.In this paper we present a method to calibrate and extract aset of key-frames from a video sequence that containenough three dimensional information to be used as thereference views for an image based reconstructionalgorithm. More specifically, the goal is to recover the 3Dgeometry of a scene from the 2D projections obtained fromthe digital images of multiple reference views, taking intoaccount the motion of the camera. Moreover, since thevideo sequence must show the object from differentperspectives, self-occlusions are permanently presentincreasing the difficulty of the problem.Inspired in [1] and [2], we present a different novelapproach based on a divide and conquer strategy to fullycalibrate a long sequence of images with a high degree offeature occlusions such as video sequences of objects inrotating platforms. The complete sequence is automaticallydivided into subsequences and, in each of them, a set ofkey-frames is selected and calibrated using an improvedversion of the algorithm presented in [5], recovering bothcamera parameters and structure of the scene. When thedifferent subsequences have been successfully calibrated amerging process groups them into a single set of camerasand reconstructed features of the scene. A final non-linearoptimization is performed in order to reduce the overallreprojection error.One advantage of the presented approach is that it allows torecover an Euclidean reconstruction of the scene withoutany initial solution or prior information, which is one of thedrawbacks of most of the existing methods. Anotherimportant feature is that the entire calibration process isbased on solving linear systems using the SVDdecomposition algorithm. The knowledge of the geometricmeaning and rank properties of the differenttransformations represented by the matrices of the processallows to enforce a valid Euclidean reconstruction. Theproposed solution is designed to be versatile in respect tothe input data allowing the use of (1) automatically trackedvideo sequences, (2) manually tracked sequences whichusually contain less frames or (3) a set of still images withfeatures and correspondences manually selected. Here on,we assume that the input data is given as a sequence ofimages and a list of features in each image and thecorrespondences with the rest of the frames. SEQUENCE FRAGMENTATIONThe fragmentation algorithm starts with the set of featuresof frame i=0 and keeps track of them in the subsequentframes until one of the following is satisfied:CAMERA CALIBRATION OF LONG IMAGE SEQUENCES WITH THE PRESENCE OF OCCLUSIONSMiguel Sainz  Antonio Susin  Nader Bagherzadeh  Image Based Modeling and Rendering Lab.                            Dynamic Simulation Lab.                 Image Based Modeling and Rendering Lab.Dept. of Electrical and Computer Science                         Dept. Matematica Aplicada 1                Dept. of Electrical and Computer ScienceUniversity of California, Irvine, USA                  Universitat Politecnica de Catalunya, SPAIN     University of California, Irvine, USA msainz@ece.uci.edu                                            toni.susin@upc.es                                          nader@uci.edu       • A minimum of key frames has been selected.• The end of the sequence is reached.• More than a user selected percentage of the original features are lost.When this occurs, a new subsequence is created as the setof key-frames starting from the first frame to the lastkeyframe before the ending condition has been triggered. At the same time, each frame is tested against the lastfound key-frame for a planar homography fit in order todetermine how much three dimensional variation ispresent. Typically the homography error is small whenthere is little variation in the camera positions between thetwo frames. We use a RANSAC based approach todetermine the percentage of inliers of the 2D homographybetween the two frames [3]. If this value is below a userselected threshold, it means that there exists somesignificant camera motion not modeled by the 2Dhomography, and the frame is marked as a key-frame.Otherwise, the frame is discarded and we proceed to thenext frame.To guarantee connectivity between the differentsubsequences so they can be merged, the last key-frame ofa subsequence is the first key-frame of the following one.FRAGMENT CALIBRATIONOnce a subsequence has been determined, we proceed toperform a complete metric calibration by extracting themeasurements that appear in all the frames of the fragmentinto a measurement matrix W. The presented calibrationsolution is a stratified reconstruction based on linearfactorization algorithms with a non linear optimizationprocess to reduce the overall reprojection error. Moreover,a robust statistical 3D analysis based on RANSAC [3] isused to improve the quality of the reconstructions.Inlier determinationDue to limitations and errors of the tracking algorithms, notall the features contained in the initial full measurementmatrix are suitable to be used for the reconstruction.Therefore an initial filtering based on a RANSAC-basedrandom sampling approach is needed in order to extract theset of inlying measures.Using this method we randomly select sets of four features,which is the minimum requirement to obtain a projectivereconstruction. A solution of P is calculated for each setand the rest of the points are obtained as a least squaresolution for P and the measurements. Then all thereconstructed points can be classified in inliers and outliersdepending on the 2D reprojection error. The solution withthe largest number of inliers will be kept.However, the proposed projective reconstruction algorithmis a robust but slow iterative approach and it is not suitableto be used multiple times as required by the RANSACfiltering. To accelerate the selection of the inliers an affinereconstruction is performed based on the randomly selectedsets which has a closed form solution [3],[4]. Once an initial set of inliers has been determined, animproved projective reconstruction is computed byiteratively reevaluating the set of inliers and calculating anew projective solution with the new set of inliers until thenumber of inlying measures remains constant. Usually thisrobust estimate converges in less that ten iterations,allowing the use of the most costly projectivereconstruction algorithm.Projective ReconstructionThe projective factorization method is a generalization ofthe factorization method which was first developed in [4],for the orthographic and the paraperspective projectionmodels respectively. It provides a more general frameworkto recover 3D shape and motion from multiple viewimages. Let Xj = (Xj, Yj, Zj , 1)T, j = 1, . . . , n, be the unknownhomogeneous 3D point vectors, Pi, i = 1, . . . , m theunknown 3 x 4 image projections, and xij = (uij, vij , 1) themeasured homogeneous image point vectors. We callprojective depths the non-zero scale factors λij relating theworld points and its projections Eq (1)The matrix W has to be a rank-4 matrix if it is the matrixassociated to a projection of a set of real points.Consequently, for points in general positions, a rank-4factorization of the scaled matrix produces a projectivereconstruction of the points.An iterative algorithm presented in [5] is used to perform arank 4 decomposition while determining the best set ofprojective depths for that factorization. Metric UpgradeWe want to update our projective scene reconstruction to ametric one. Essentially the autocalibration process can besummarized into the problem of computing a projectivedistortion matrix (PDM), in other words, an homography Hsuch that,  and  are the metricreconstruction of the scene. The metric camera matricescan be decomposed in terms of the internal and externalparameters asEq (2)where the 3 x 3 symmetric matrix Ki is the internal cameraparameters matrix, Ri is the euclidean rotation matrix and tiis the translation vector.To derive the auto-calibration equations [3], [7], let usWλ11x11 … λ1nx1n… … …λm1xm1 … λmnxmnP1…PmX1 … Xn= =PiM PiH= XjM H 1– Xj=PiM Ki Ri ti[ ] i, 1 …m,= =choose the world coordinate frame aligned with the firstcamera. If we write the plane at infinity as an usual plane,, thenEq (3)To derive the auto-calibration equations, first we expressthe rest of the projective cameras distinguishing betweenthe first three and the last columns, , and fromequations (1) and (2) we obtain (see [3])Eq (4)The symmetric matrix  is known to be the dual imageof the absolute conic, , which is related with theabsolute dual quadric, , by Eq (5)As we are using the first camera frame as the origin, we canwriteEq (6)A linear system can be obtained from (6) and (5) assumingsome knowledge on the camera internal parameters. Forinstance, if the principal point is at the origin then. From (5) two linear equations can bederived for the entries of . When, in addition, zero skewis assumed another linear relation is added .Finally, if an equal aspect ratio is assumed a last linearequation  can be added. This means that 4 linearequations can be derived from every frame when theprevious assumptions are considered. When looking at this assumptions gives (  is symmetrical)Eq (7)The autocalibration equations (5) becomes anoverdetermined linear system of 4 x m equations and onlyfive unknowns. A supplementary non linear condition must be added to the least square solution.If we express the solution of the overdetermined system asa linear combination of the solution vector and a constant λby the eigenvector with the lowest eigenvalue, a thirddegree polynomial in terms of λ can be obtained to enforcethe supplementary condition. Then, using SVD to obtain Hand back-substituting in the equations, a final metricreconstruction is computed under the assumption of knownprincipal points and skew values.OptimizationThe solution presented above is a closed form least squaredconstrained approximation of the structure from motionproblem. A final non-linear optimization process isrequired in order to reduce the reprojection erroraccounting for all the nonlinearities not recovered in themetric solution. Moreover, if a more sophisticated camerainternal parameter description is required, it can beincorporated into the optimization process as well.Additionally, since the cameras have been recovered,several of the features not included in the reconstruction,because they are not present in all the frames, can berecovered by least squared approximation, increasing thenumber of reconstructed points.A bundle adjustment is then computed using the standardsparse Levenberg-Marquardt algorithm described in theliterature (e.g. in [3] or [8]), using the reconstruction as theinitial solution.SEQUENCE MERGINGThe merging of two subsequences is performed in metricspace by determining the set of common points, betweenthe last frame of one sub-sequence and the first of the nextone, that by construction correspond to the same cameraand measurements.Since both subsequences have been calibrated, the set ofreconstructed common points must be similar up to ascaling and a translation assuming both frames alignedwith the world reference. An overdetermined linearequation system can be built relating the reconstructedfeatures and solved using SVD. Then, the scaling andtranslation transformation is applied and the sub-sequencesbecome merged.Due to noise related errors during calibration or becausethe set of features used in each of the subsequences isdifferent, small disparities in the focal length between thetwo overlapped frames can appear. This does not affect theoverall reprojection error since it is a reconstruction fullycompatible with the input data. In the case that a correctionwas necessary, a recalibration of one of the subsequencesshould be performed using the desired focal length as aconstraint. However for the purpose of the presented workthis is not required since the 3D reconstruction of thefeatures is accurate and the reprojection error is small.Once all the subsequences have been merged, a finalLevenberg-Marquardt adjustment is computed over thecomplete sequence to reduce the overall error.RESULTSSeveral validation and verification tests have beenperformed with real imagery. The two datasets presented inthis paper consist of video sequences of a circular cameramotion around a set of objects. All tests were performed ona PC system with a 2.2Ghz P4, 1Gb of RAM running MSWindows XP. π∞ pT 1( , )T=H K1 0pTK1– 1=Pi Ai ai ][=KiKiT Ai aipT–( )K1K1T Ai aipT–( ) i, 2…m= =KiKiTω*iQ∞*ω*i PiQ∞* PiT=Q∞*K1K1T K1K1Tp–pTK1K1T– pTK1K1Tpω*1 ω– *1pp– Tω*1 pTω*1p= =ω13*i ω23*i 0= =Q∞*ω12*i 0=ω11*i ω22*i=Q∞*Q∞*Q∞*q11 0 0 q140 q11 0 q240 0 1 q34q14 q24 q34 q44HHT= =det Q∞*( ) 0=The first dataset, monster, consists of 423 frames and wascaptured using an on-the-shelf digital video camcorder at720x480 pixels of resolution. The object is a 20cm tallconcrete statue on top of a turning table. The camera wasset to autofocus and the object was manually rotated.Several markers where located on the surface of the objectand a semi-automatic tracking tool was used to generatethe measurement data.The second dataset, house, is the well know sequence fromthe project MOVI and consists of 114 frames. Themeasurement data was obtained using the KLT tracker [6]set to track 500 features per frame replacing the ones thatare lost due to noise or occlusions. Some numerical results are presented in Table 1 showingthe reconstruction speed and how much time each of theparts of the method use. The most expensive is the finalbundle adjustment because it uses all the selected framesand all available features. The final reprojection meanerrors are very low (below 1 pixel) showing the accuracyof the calibration method.On top of the page, two figures show the 3D reconstructionof the features and the cameras as well as one of the framesof each sequence.CONCLUSIONSIn this paper we have presented a novel divide-and-conquerapproach to the problem of calibrating a moving cameraduring long sequences with feature occlusions, with thepurpose of identifying and recover 3D information to beused in an image based modeling tool.An automatic key-frame selection and sub-sequenceconstruction strategy is used to select and group thoseframes from the sequence that most likely containsignificant three dimensional information. A fast and robust calibration tool is used to recover thestructure and motion from each of the subsequences. Thisalgorithm is mainly linear in the unknowns and does notrequire any prior knowledge of the scene or cameraparameters.The extensive use of RANSAC-based techniques, both in2D and 3D space, provide extra robustness to thereconstruction algorithms. Moreover the proposed methodis computationally fast in standard PC computers, makingit a very attractive solution.Acknowledgments. This research was supported by theNational Science Foundation under contract (CCR-0083080) and by the Comissio Interdepartamental eRecerca i Innovacio Tecnologica,(CIRIT), Caspar dePortola grant C02-03.REFERENCES[1] Fitzgibbon A. and Zisserman A. Automatic camera recovery forclosed or open image sequences. In Proc. European Conference onComputer Vision, pages 311ñ326. Springer-Verlag, June 1998.[2] Gibson S., Cook J., Howard T.L.J., Hubbold R.J., and Oram D.Accurate Camera Calibration for Off-line, Video-Based AugmentedReality. IEEE and ACM International Symposium on Mixed andAugmented Reality, Darmstadt, Germany, September 2002. [3] Hartley R., Zisserman A., Multiple View Geometry in ComputerVision. Cambridge Univ. Press, 2000.[4] Poelman C.J. and Kanade T., A paraperspective factorization methodfor shape and motion recovery Technical Report CMU-CS 93-219,Carnegie Mellon University, December 1993.[5] Sainz M., Bagherzadeh N. and Susin A., Recovering 3D MetricStructure and Motion from Multiple Uncalibrated Cameras. In IEEEProc. International Conference on Information Technology: Codingand Computing, pp 268-273, 2002.[6] Tomasi C. and Kanade T. Detection and tracking of point features.Technical Report CMU-CS-91-132, Carnegie Mellon University,April 1991.[7] Triggs B., Autocalibration and the Absolute Quadric IEEE CVPR96,845-851, 1996.[8] Triggs B.,  McLauchlan P.F., Hartley R.I., and Fitzgibbon A.W.Bundle adjustment - a modern synthesis. In B. Triggs, A. Zisserman,and R. Szeliski (eds.), Vision Algorithms: Theory and Practice, pp.298--472. Springer-Verlag, 2000.# of Frames # of used frames# of featuresTotal time(sec)Selection time (sec.)Calibration time (sec.)Bundle time (sec.)# sub sequencesMean error(pixels)ErrorVar.Monster 423 68 52 130.1 5.1 2.8 122.1 7 0.64 0.58House 114 114 936 371.5 23.9 127.7 219.7 12 0.81 0.96Table 1: Statistics of the datasets

Camera calibration of long image sequences with the presence of occlusions

http://citeseerx.ist.psu.edu/viewdoc/summary?doi=10.1.1.104.8053

Camera calibration of long image sequences with the presence of occlusions

Abstract

Similar works

Full text

Available Versions

UPCommons. Portal del coneixement obert de la UPC