Rectification of a stereo pair from the fundamental matrix (also known as uncalibrated rectification [1]) is a procedure
that results in a new pair of images that have strictly horizontal epipolar lines. Moreover, the corresponding epipolar lines
of the two images acquire the same vertical position. This means that the vertical disparity between corresponding pixels
becomes small enough (e.g. zero, if we speak about integer pixel coordinates), and efficient stereo matching algorithms
can be applied further. This article presents a plausible algorithm to achieve this. The plausibility consists in the fact
that the algorithm works not for all cases, and these cases are studied in detail in this paper. However, authors claim
that the presented algorithm deserves attention because of implementation and explanation simplicity

Krygin, V.

Nechaieva, V.

Electronic Archive of Kyiv Polytechnic Institute

UDC 514.123.1PLAUSIBLE RECTIFICATION FROM THE FUNDAMENTAL MATRIXV. Nechaieva1, a, V. Krygin2,3, b1Taras Shevchenko National University of Kyiv2National Technical University of Ukraine «Igor Sikorsky Kyiv Polytechnic Institute»,Institute of Physics and Technology3 International Research and Training Center for Information Technologies and Systems of the NationalAcademy of Sciences of Ukraine and Ministry of Education and Science of UkraineAbstractRectification of a stereo pair from the fundamental matrix (also known as uncalibrated rectification [1]) is a procedurethat results in a new pair of images that have strictly horizontal epipolar lines. Moreover, the corresponding epipolar linesof the two images acquire the same vertical position. This means that the vertical disparity between corresponding pixelsbecomes small enough (e.g. zero, if we speak about integer pixel coordinates), and efficient stereo matching algorithmscan be applied further. This article presents a plausible algorithm to achieve this. The plausibility consists in the factthat the algorithm works not for all cases, and these cases are studied in detail in this paper. However, authors claimthat the presented algorithm deserves attention because of implementation and explanation simplicity.Keywords: fundamental matrix, projective rectification, stereo vision, epipolar geometry, computer visionIntroductionThe stereo correspondence problem is complex. Oneof the possible ways to state it is: how to find thecorrespondence between pixels of both images so thatthey are the projections of the same points of observedobjects? It means that each pixel of one image islabelled either by the coordinate of the correspondingpixel of another image or by “no correspondence” label.Formally,𝑑 : 𝐼𝐿 → 𝐼𝑅 ∪ {∅} ,where 𝐼𝐿 is a set of the left image pixels and 𝐼𝑅 is a setof the right image pixels. So, we look for an answer inthe set of |𝐼𝐿||𝐼𝑅|+1 possible ones. A good rectificationcan collapse the answers set dramatically: given theright image of width 𝑤 and height ℎ, we can reducethe |𝐼𝐿||𝐼𝑅|+1 = |𝐼𝐿|𝑤·ℎ+1 to |𝐼𝐿|𝑤+1 by removing thevertical disparity. The stereo correspondence itself isnot a subject of this paper, but the interested reader isreferred to [2, 3].In the first section of this paper, we describe the rec-tification of the right image, provided by [4]. We’re notdiving into details of epipolar geometry – just providingthe main definitions and relations.In the second section, we describe our plausiblemethod of rectification of the left image based on therectification of the right one.The third section provides an analysis of the domainof the proposed algorithm: the cases when it doesn’twork.averonika.netch@gmail.combvaleriy.krygin@gmail.com1. Right image rectificationWe are looking for matrices that transform the im-ages so that their corresponding pixels lie on the samehorizontal lines. One instrument that is going to helpus is the fundamental matrix 𝐹 ∈ R3×3.The fundamental matrix specifies the relation be-tween the points of two images, lying on the corre-sponding epipolar lines𝑥′𝑇 · 𝐹 · 𝑥 = 0.Given a set of corresponding points 𝐶 ⊂ 𝐼𝐿 × 𝐼𝑅 (canbe obtained with the help of [5, 6, 7]), the fundamentalmatrix can be estimated using the 8-point algorithm,7-point algorithm [8], or any other.The fundamental matrix 𝐹 has rank 2 so its leftand right kernels are one-dimensional spaces. A vector𝑒 =[︀𝑒𝑥 𝑒𝑦 1]︀𝑇 from the right kernel is called theepipole of the left image. Geometrically, epipole of animage is a projection of an optical center of anothercamera. In the same manner, 𝑒′ =[︀𝑒′𝑥 𝑒′𝑦 1]︀𝑇 fromthe left kernel is called the epipole of the right image.Henceforth, we will use 𝑒 and 𝑒′ for the unit vectors,having the same directions as the epipoles. This re-definition simplifies further formulas but doesn’t breakthem.We assume that we have a pinhole camera with animage center in[︀0 0 1]︀𝑇 and the camera center in[︀0 0 0]︀𝑇 . To make the epipolar lines horizontal, wemust move the epipolar point to[︀1 0 0]︀𝑇 . This canbe performed with a linear mapping called rectification.Let us call this mapping 𝑅𝑟, or the rectification matrix𝑅𝑟 = 𝐺 ·𝑅,Математичнi методи комп’ютерного моделювання та кiбернетичної безпеки101where 𝑅 is a rotation matrix – it rotates the image sothat the epipole lies on the abscissa of the image, and 𝐺is a shear matrix – it makes the epipolar lines parallelto each other, i.e., it moves the epipolar point to theright into the infinity.Let us look a little further into these components.While there are different ways to represent them [4, 9],we choose the following way. The 𝑅 matrix consists oforthonormal vectors𝑅 =⎡⎣(𝑘 × (𝑘 × 𝑒′))𝑇(𝑘 × 𝑒′)𝑇𝑘𝑇⎤⎦ ,where 𝑘 =[︀0 0 1]︀𝑇 is one of the basis vectors of thespace. The 𝐺 matrix is𝐺 =⎡⎣1 0 00 1 0−𝑒′𝑧𝑥 0 1⎤⎦ ,where𝑥 = (𝑅 · 𝑒′)𝑥 = (𝑘 × (𝑘 × 𝑒′))𝑇 · 𝑒′. (1)Thus, 𝑅𝑟 is𝑅𝑟 = 𝐺 ·𝑅 =⎡⎢⎣(𝑘 × (𝑘 × 𝑒′))𝑇(𝑘 × 𝑒′)𝑇(𝑘 × (𝑘 × 𝑒′))𝑇 · −𝑒′𝑧𝑥 + 𝑘𝑇⎤⎥⎦ .You can read more about these implications in [4, 8].You can see a comparison of one of our experimentalrectification results with the one achieved with OpenCV[10] on fig. 1. Fig. 1b and fig. 1a are the inputimages taken by a smartphone. Correspondences andthe fundamental matrix were found by an application,written for this paper evaluation. Fig. 1d and fig. 1c arethe results of applying stereoRectifyUncalibratedfunction from OpenCV library. Fig. 1f and fig. 1eare the results of applying the plausible rectificationmethod. You can see that both methods give similarresults.The pairs of rectification results are valid stereograms:you can cross your eyes to see the third picture betweenthem and focus on the paper plane to percept the depth.The digital version of the article contains high-qualitypictures, so it’s recommended to perform the depthperception with it.2. Plausible rectification of the left im-ageWe will use a matrix[𝑒′]× =⎡⎣0 −𝑒′𝑧 𝑒′𝑦𝑒′𝑧 0 −𝑒′𝑥−𝑒′𝑦 𝑒′𝑥 0⎤⎦ .It can be trivially checked that[𝑒′]× · 𝑥 = 𝑒′ × 𝑥, ∀𝑥 ∈ R3.Let us build the intermediate matrix of rank 2𝑀 = 𝑅𝑟 · [𝑒′]× · 𝐹and denote its last two rows as 𝑚𝑇𝑦 and 𝑚𝑇𝑧 , andconstruct the new matrix𝑀 ′ =⎡⎢⎣(𝑚𝑦×𝑚𝑧)𝑇√‖𝑚𝑦×𝑚𝑧‖𝑚𝑇𝑦𝑚𝑇𝑧⎤⎥⎦ .Usually, x-coordinates of the left image, resampled withthe 𝑀 ′ matrix, differ a lot from the correspondingones of the transformed right image. That’s why it’srecommended to use one more trick [4] and define theleft image rectification matrix as𝑅𝑙 =⎡⎣𝑎 𝑏 𝑐0 1 00 0 1⎤⎦ ·𝑀 ′,where 𝑎, 𝑏 and 𝑐 are the solutions to the following linearleast squares problem:∑︁⟨𝑥,𝑥′⟩∈𝒞⎛⎜⎝⎡⎣𝑎𝑏𝑐⎤⎦𝑇· 𝑀′ · 𝑥(𝑀 ′ · 𝑥)𝑧− (𝑅𝑟 · 𝑥′)𝑥(𝑅𝑟 · 𝑥′)𝑧⎞⎟⎠2→ min𝑎,𝑏,𝑐∈R,where 𝒞 is a set of corresponding points, found for thefundamental matrix estimation. Don’t forget to divideby 𝑧-coordinates to normalize coordinates of the pixels.The next section provides a derivation of the algo-rithm and analysis of its weaknesses.3. Analysis of the plausible rectificationmethodLet 𝑀 be the intermediate epipolar projection matrixthat transforms the left image. The fundamental matrix𝐹 can be represented as 𝐹 = [𝑒′]× · 𝑆, where 𝑆 is atransformation from the space of the right image to thespace of the left one. One of the methods to fetch thismatrix (up to scale) is [11]𝑆 = [𝑒′]× · 𝐹.This means that the linear transformation 𝑀 consistsof two parts: teleport the left image to the right imagespace using 𝑆 and rectify it there using 𝑅𝑟 matrix𝑀 = 𝑅𝑟 · 𝑆= 𝐺 ·⎡⎢⎣((𝑘 × (𝑘 × 𝑒′))× 𝑒′)𝑇((𝑘 × 𝑒′)× 𝑒′)𝑇(𝑘 × 𝑒′)𝑇⎤⎥⎦ · 𝐹= 𝐺 ·⎡⎢⎣(𝑘 × 𝑒′)𝑇((𝑘 × 𝑒′)× 𝑒′)𝑇(𝑘 × 𝑒′)𝑇⎤⎥⎦𝑇· 𝐹.As we can see, the rank of this matrix is not greaterthan 2. We want it to become full-rank after we replacethe first row of it. For this, we have to make sure thatthe second and third rows are linearly independent.Let us look into 𝐺 ·𝑅 · [𝑒′]× first. Since(𝑘 × (𝑘 × 𝑒′))× 𝑒′ = 𝑘 × 𝑒′,Всеукраїнська науково-практична конференцiя студентiв, аспiрантiв та молодих вчених102(a) Original right image (b) Original left image(c) The right image rectified using OpenCV (d) The left image rectified using OpenCV(e) The right image rectified using our plausible method (f) The left image rectified using our plausible methodFig. 1. Experimental resultswe get𝐺 ·𝑅 · [𝑒′]× =⎡⎢⎣(𝑘 × 𝑒′)𝑇(𝑘 × (𝑘 × 𝑒′))𝑇(𝑘 × 𝑒′)𝑇 ·(︁−𝑒′𝑧𝑥 + 1)︁⎤⎥⎦ .Since 𝑘 × (𝑘 × 𝑒′) ⊥ (𝑘 × 𝑒′) for any non-zero vectors,the only case that worries us is when one of the rowsequals zero. The second line can turn into zero only if𝑘 and 𝑒′ are collinear. That would mean that 𝑒′ liesexactly in the center of the image. If 𝑒′ lies within animage, then it is impossible to make all the epipolarlines parallel, and 𝑥 = 0, so 𝐺 does not even exist; sothis case is not a subject of the paper.The only edge case that does interest us is−𝑒′𝑧𝑥= −1.Substitution of 𝑥 with (1) gives𝑒′𝑧 = (𝑘 × (𝑘 × 𝑒′))𝑇 · 𝑒′.Since 𝑘 is the applicate, 𝑒′𝑧 = 𝑘 · 𝑒′. Then we get𝑘𝑇 · 𝑒′ = (𝑘 × (𝑘 × 𝑒′))𝑇 · 𝑒′. (2)Let us say that 𝛼 is the angle between 𝑒′ and 𝑘. Since𝑘, 𝑒′, and 𝑘 × (𝑘 × 𝑒′) are coplanar, and 𝑘 × (𝑘 × 𝑒′)is perpendicular to 𝑘, 𝑒′ can be written as a linearcombination of 𝑘 × (𝑘 × 𝑒′) and 𝑘:𝑒′ = cos𝛼 · 𝑘 + sin𝛼 · 𝑘 × (𝑘 × 𝑒′)Since 𝑒′ is not collinear to 𝑘, we know for sure thatsin𝛼 ̸= 0 and can divide by it𝑘 × (𝑘 × 𝑒′) = 𝑒′ − cos𝛼 · 𝑘sin𝛼and (2) turns into𝑘𝑇 · 𝑒′ = 𝑒′𝑇 · 𝑒′ − cos𝛼 · 𝑘𝑇 · 𝑒′sin𝛼,implying(cos𝛼+ sin𝛼) · 𝑘𝑇 · 𝑒′ = 𝑒′𝑇 · 𝑒′.Математичнi методи комп’ютерного моделювання та кiбернетичної безпеки103By definition of the dot product,(cos𝛼+ sin𝛼) · ‖𝑘‖ · ‖𝑒′‖ · cos 𝑎 = ‖𝑒′‖2All the vectors are of length 1, meaning(cos𝛼+ sin𝛼) · cos𝛼 = 1,thussin𝛼 · (sin𝛼− cos𝛼) = 0,which means the following solutions for 𝛼:𝛼 = 𝜋 · 𝑛,𝛼 = 𝜋 · 𝑛− 𝑝𝑖4,∀𝑛 ∈ Z.The first case means that 𝑒′ is collinear to the applicate,but we have already omitted this case. So, the onlysituation that troubles us is when 𝑒′ creates a 45∘ anglewith one of these two vectors and a 135∘ with anotherone. This is a significant problem with the method.Now we return to the original formula𝑀 = 𝐺 ·𝑅 · [𝑒′]× · 𝐹and see what happens if we unfold it completely.𝑀 =⎡⎢⎣(𝑘 × 𝑒′)𝑇 · 𝐹(𝑘 × (𝑘 × 𝑒′))𝑇 · 𝐹(𝑘 × 𝑒′) · 𝐹 ·(︁−𝑒′𝑧𝑥 + 1)︁⎤⎥⎦ ,where 𝐹 is the fundamental matrix, which has rank 2.The only unit vector in its left nullspace is 𝑒′. Considerthe decomposition of an arbitrary vector 𝑥 into𝑥 = 𝑦 + 𝑐 · 𝑒′, 𝑐 ∈ R, 𝑦 ∈ R3, 𝑦 ⊥ 𝑒′.As long as 𝑒′ belongs to the left kernel of the funda-mental matrix 𝐹 , meaning 𝑐 · 𝑒′𝑇 · 𝐹 = 0, the followingequation applies:𝑥𝑇 ·𝐹 = (𝑦 + 𝑐 · 𝑒′)𝑇 ·𝐹 = 𝑦𝑇 ·𝐹 + 𝑐 ·𝑒′𝑇 ·𝐹 = 𝑦𝑇 ·𝐹,Thus, for any 3D vectors 𝑥 and 𝑦𝑥𝑇 · 𝐹 = 𝑦𝑇 · 𝐹 ⇐⇒ ∃𝑐 ∈ R : 𝑥 = 𝑦 + 𝑐 · 𝑒′, 𝑒′ ⊥ 𝑦.Suppose that multiplying by 𝐹 makes the second andthird rows’ vectors collinear. It would mean that𝑘 × 𝑒′ = 𝑘 × (𝑘 × 𝑒′) + 𝑐 · 𝑒′.This case is impossible since 𝑘 × 𝑒′ ⊥ 𝑘 × (𝑘 × 𝑒′) and𝑘 × 𝑒′ ⊥ 𝑒′ if 𝑒′ ̸= 𝑘. It means that the fundamentalmatrix does not create any more special cases, and thusthe limitations stay the same.To conclude, we have three different cases where themethod fails:∠ (𝑘 × (𝑘 × 𝑒′) , 𝑒′) = 135∘ & ∠(𝑘, 𝑒′) = 45∘;∠ (𝑘 × (𝑘 × 𝑒′) , 𝑒′) = 45∘ & ∠(𝑘, 𝑒′) = 135∘;∠(𝑘, 𝑒′) = 0∘,(3)where the center of the image lies in 𝑘.4. Further workFurther work can be divided into two parts. First,some cases change the flow of the analysis – for exam-ple, if another method of right image rectification isused. Second, it is possible that in the cases where theplausible rectification fails, certain alternations of themethod may be used.ConclusionThe plausible rectification method has a certain el-egancy and does not require too many calculations.However, the method is not universal; it fails when thebasis vector 𝑘 of the space creates a 0∘, 45∘ or 135∘angle with the epipole (3). The number of such casesis limited but still can affect the method correctnessdepending on the situation.References1. Fusiello Andrea, Irsara Luca. Quasi-Euclidean un-calibrated epipolar rectification. –– 2009. –– 01. ––P. 1 – 4.2. Savchynskyy Bogdan. Discrete Graphical Models— An Optimization Perspective // Foundationsand Trends® in Computer Graphics and Vision. ––2019. –– Vol. 11, no. 3-4. –– P. 160–429.3. Szeliski Richard. Computer vision algorithms andapplications. –– London; New York : Springer,2011. –– Access mode: http://dx.doi.org/10.1007/978-1-84882-935-0.4. Hartley Richard I. Theory and Practice of Pro-jective Rectification // Int. J. Comput. Vis. ––1999. –– Vol. 35, no. 2. –– P. 115–127. –– Access mode:https://doi.org/10.1023/A:1008115206617.5. Lowe David G. Object Recognition from LocalScale-Invariant Features // Proceedings of the Inter-national Conference on Computer Vision-Volume 2- Volume 2. –– ICCV ’99. –– USA : IEEE ComputerSociety, 1999. –– P. 1150.6. Speeded-Up Robust Features (SURF) / Her-bert Bay, Andreas Ess, Tinne Tuytelaars,Luc Van Gool // Comput. Vis. Image Underst. ––2008. –– Jun. –– Vol. 110, no. 3. –– P. 346–359. ––Access mode: https://doi.org/10.1016/j.cviu.2007.09.014.7. ORB: An Efficient Alternative to SIFT or SURF /Ethan Rublee, Vincent Rabaud, Kurt Konolige,Gary Bradski // International Conference on Com-puter Vision. –– Barcelona, 2011. –– 11/2011.8. Hartley R. I., Zisserman A. Multiple View Geome-try in Computer Vision. –– Second edition. –– Cam-bridge University Press, ISBN: 0521540518, 2004.9. Mallon John, Whelan Paul F. Projective rectifica-tion from the fundamental matrix // Image and Vi-sion Computing. –– 2005. –– Vol. 23, no. 7. –– P. 643 –650. –– Access mode: http://www.sciencedirect.com/science/article/pii/S0262885605000399.10. Bradski G. The OpenCV Library // Dr. Dobb’sJournal of Software Tools. –– 2000.11. Luong Q.-T., Viéville T. Canonical Representa-tions for the Geometries of Multiple ProjectiveViews // Computer Vision and Image Understand-ing. –– 1996. –– Vol. 64, no. 2. –– P. 193 – 229. ––Access mode: http://www.sciencedirect.com/science/article/pii/S1077314296900557.Всеукраїнська науково-практична конференцiя студентiв, аспiрантiв та молодих вчених104

Plausible rectification from the fundamental matrix

https://ela.kpi.ua/bitstream/123456789/50771/1/%28101-104%29_Nechaieva.pdf

Plausible rectification from the fundamental matrix

Abstract

Similar works

Full text

Available Versions

Electronic Archive of Kyiv Polytechnic Institute