The estimation of parametric global motion has had a significant attention
during the last two decades, but despite the great efforts invested, there
are still open issues. One of the most important ones is related to the ability to recover
large deformation between images in the presence of illumination changes
while kipping accurate estimates. Illumination changes in color images are another
important open issue. In this paper, a Generalized least squared-based motion
estimator is used in combination with color image model to allow accurate
estimates of global motion between two color images under the presence of large
geometric transformation and illumination changes. Experiments using challenging
images have been performed showing that the presented technique is feasible
and provides accurate estimates of the motion and illumination parameter

A. Bartoli

B. Zitova

D.G. Lowe

F. Mindru

G. Healey

G. West

G. Wyszecki

G.D. Finlayson

L. Shao

P.H.S. Torr

R. Montoliu

R.C. Gonzalez

S. Kaneko

Montoliu Colás, Raul

Latorre Carmona, Pedro

Pla, Filiberto

Repositori Institucional de la Universitat Jaume I

Color Image Registration Under Illumination Changes ?Rau´l Montoliu1, Pedro Latorre Carmona2, and Filiberto Pla21 Dept. Arquitectura y Cie´ncias de los Computadores.2 Dept. Lenguajes y Sistemas Informa´ticos.Jaume I UniversityCampus Riu Sec s/n 12071 Castello´n, Spain[montoliu,latorre,pla]@uji.es, http://www.vision.uji.esAbstract. The estimation of parametric global motion has had a significant at-tention during the last two decades, but despite the great efforts invested, thereare still open issues. One of the most important ones is related to the ability to re-cover large deformation between images in the presence of illumination changeswhile kipping accurate estimates. Illumination changes in color images are an-other important open issue. In this paper, a Generalized least squared-based mo-tion estimator is used in combination with color image model to allow accurateestimates of global motion between two color images under the presence of largegeometric transformation and illumination changes. Experiments using challeng-ing images have been performed showing that the presented technique is feasibleand provides accurate estimates of the motion and illumination parameters.1 IntroductionImage registration ([17]) could be defined as the process to transform an image so thatit matches another image as correctly as possible. This process is necessary when wewant to compare or to integrate the data information from the two images. In the im-age acquisition process of a scene there are many factors involved: the position andthe distance from the camera (or sensor) to the scene, the illumination, the nature ofthe objects to be imaged, etc. Any change in these factors implies that the data in thecorresponding images are not directly comparable.During the last few years, a special interest has emerged in relation to the needto cope with simultaneous viewpoint and illumination changes ([10], [12], [1], [2], tocite a few works). Important examples are those related to the retrieval of images inimage databases where any user can upload an image of a famous place, and whereeach person may use different types of cameras and acquire the scene from differentconfigurations and under different illumination conditions (cloudy, sunny, etc.).There are evidently different methodologies to assess the best transformation be-tween two images. One of them considers different versions of optimization-based mo-tion estimation methods. Their main advantage is that the motion parameter estimationprocess is deeply over-constrained. Nevertheless, these methods suffer from a series ofdrawbacks [11]. One of the most important drawbacks is the presence of outliers, in theform of occlusions due to motion, to sensor noise, or to illumination changes.? This work has been partially funded with project X from the Y.In general, we may consider the direct geometric registration problem as that ofminimizing an error function in relation to the difference in the pixel values between animage we may call Test image and the Reference image. In particular, it can be formallywritten as:ming∑q∈<‖I1(qi)− I2(G(qi;g))‖2 (1)where I1 and I2 are two input images, qi = (xi, yi)T are the pixel coordinates, g isthe vector of motion parameters and G is the function to transform the pixel coordinatesfrom one image to the other. The function G is expressed, for instance, in an affine (Eq.2) motion (Eq. ??) as follows:G(qi;g) =(a1xi + b1yi + c1a2xi + b2yi + c2)(2)If we also consider photometric changes, these may be modeled by a transformationP with parameter vector p and the minimization would therefore be:ming∑q∈<‖I1(qi)− P(I2(G(qi;g));p)‖2 (3)To solve the problem shown in Eq. 3, Bartoli developed the Dual Inverse Com-positional (DIC) estimation technique [2] where he considers Eq. 3 and then appliesan inverse compositional update rule for both the geometric and photometric transfor-mations. See [2] for details on the steps of the algorithm used to assess the geometricregistration and illumination compensation parameters.In this paper a generalized least squares-based non-linear motion estimation tech-nique that incorporates the capability to deal with color illumination changes is pre-sented, where illumination changes are modeled considering an affine transformationframework. The method is based on the GLS motion estimation method introduced byMontoliu and Pla in [11], and where a new set of functionals is proposed, incorporatingthese illumination changes. GLS method is applied on the functionals, deriving a newset of equations that allow the simultaneous assessment of the geometric and affine illu-mination transformation parameters. We show that the method we propose gives betterresults than the method recently described in [2].The rest of the paper is organized as follows: section 2 justify bla bla. In Section?? the GLS for general problems is briefly introduced. Section 4 presents our proposedmethod. Section 5 shows the experiments and results and finally we conclude in Section6.2 Illumination compensation modelIllumination compensation is closely related to what it is called chromatic adaptation inhuman colour vision. The first chromatic adaptation experiments started in the late 40sof the last century. A few years later, in a experiment made by Wyszecki and Stiles onhuman asymmetric matching [15], a human subject had different adaptation states ontwo different parts of his visual field. The subject viewed a test light in one adaptationstate and adjusted a mixture of three primary lights in the other adaptation state untilthe test light matched the mixture. They proved that a diagonal linear matrix transformwould be enough to reproduce the experiment of asymmetric matching. Nevertheless,West and Brill [14] and others proved later that for a given set of sensor sensitivities adiagonal transform could only cover a restricted group of object colours and illuminantspectra.Finlayson et al [3] reasoned that a diagonal transform would be enough for themodeling of an illumination change if the we may have extremely narrow-band sensorsin the camera, which is often not the case. There are other cases where illuminationcompensation can fail, for instance, if there are other processes happening like bias inthe camera, or saturated colours in the scene. In the latter case, some colours wouldfall in (and outside of) the camera gamut boundary [4]. This is the reason why the useof a complete (full) affine transform in the form Ω · I(q) + Φ is justified (see, forexample, [5], [8], [16], to cite a few), where Ω ∈ <3×3 is a full matrix, with elementsωkl, k, l = 1, . . . , 3, and Φ ∈ <3, a vector with elements φk, k = 1, . . . , 3.3 GLS for general problemsIn general, the GLS estimation problem can be expressed as follows (see [11] for moredetails):minimize [Θυ = υTυ] subject to F(χ, λ) = 0, (4)where:– υ is a vector of r unknown residuals in the observation space, that is, υ = λ − λ˜,where λ and λ˜ are the unperturbed and actually measured vector of observations,respectively.– χ = (χ1, . . . , χm)T is a vector of m parameters;– λ is made up by r elements λi, λ = (λ1, . . . , λr)T , each one is an observationvector with n components λi = (λ1i , . . . , λni )T– F(χ, λ) is made up by r elementsFi(χ, λi),F(χ, λ) = (F1(χ, λ1), . . . ,Fr(χ, λr))T ,each one is, in general, a set of f functions that depend on the common vector of pa-rameters χ and on an observation vector λi,Fi(χ, λi) = (F1i (χ, λi), . . . ,Ffi (χ, λi))T .Those functions can be non-linear.Thus, the solution of (4) can be addressed as an iterative optimization starting withan initial guess of the parameters χ̂(0). At each iteration j, the algorithm estimates∆̂χ(j) to update the parameters as follows: χ̂(j) = χ̂(j − 1) + ∆̂χ(j). The processis stopped if the improvement ∆̂χ(j) at iteration j is smaller than an user-specifiedresolution in the parameter space.The improvement ∆̂χ(j) can be expressed as follows (see [11] for more details):∆̂χ(j) = (ATQA)−1ATQe, (5)where the matrix Q = (BBT )−1 has been introduced to simplify the notation. Equation5 can also be expressed in a more convenient way as follows:∆̂χ(j) =( ∑i=1...rNi)−1( ∑i=1...rTi,), (6)where Ni = Ati(BiBti)−1Ai and Ti = Ati(BiBti)−1ei, withBi =∂F1i (χ̂(j−1),λi)∂λ1i. . .∂F1i (χ̂(j−1),λi)∂λni......∂Ffi (χ̂(j−1),λi)∂λ1i. . .∂Ffi (χ̂(j−1),λi)∂λni(f×n), (7)Ai =∂F1i (χ̂(j−1),λi)∂χ1 . . .∂F1i (χ̂(j−1),λi)∂χp......∂Ffi (χ̂(j−1),λi)∂χ1 . . .∂Ffi (χ̂(j−1),λi)∂χp(f×m), (8)ei =−F1i (χ̂(j − 1), λi)...−Ffi (χ̂(j − 1), λi)(f×1). (9)4 GLS-based color motion estimation under illumination changesIn our formulation of the motion estimation problem, the function Fi(χ, λi) is ex-pressed as follows:Fi(χ, λi) = I2(G(qi;g))− P−1(I1(qi);p) (10)I1(qi) = (R1(qi), G1(qi), B1(qi))T and I2(q′i) = (R2(q′i), G2(q′i), B2(q′i))T , whereq′i has been introduced to simplify notation as: q′i = G(qi;g). Note that in this case thenumber of functions f is 3. The Eq. 10 can also be writen in a more convient way asfollows:F1i (χ, λi) = R2(q′i)− (R1(qi)ω11 +G1(qi)ω12 +B1(qi)ω13 + φ1)F2i (χ, λi) = G2(q′i)− (R1(qi)ω21 +G1(qi)ω22 +B1(qi)ω23 + φ2)F3i (χ, λi) = B2(q′i)− (R1(qi)ω31 +G1(qi)ω32 +B1(qi)ω33 + φ3)(11)whereR1(qi),G1(qi) andB1(qi) are theR,G andB, components of the first color im-age in the sequence (reference image) at the point qi, and R2(q′i), G2(q′i) and B2(q′i)are the R, G and B, components of the second color image in the sequence (test im-age) at the transformed point q′i = G(qi;g). In this case, each observation vector λi isrelated to each pixel qi, with r being the number of pixels in the area of interest.Let us define the observation vector as λi = (R1(qi), G1(qi), B1(qi), xi, yi). Thevector of parameters is defined as follows: χ = (g,p)T .Due to the big dimension of the parameter vector it is difficult to show Ai, Bi usingmatrices. We use tables intead. For afine motion, The Ai matrix is showed in two tables:1 and 2; Bi is showed in two tables: 3 and 4. For projective motion, The Ai matrix isshowed in two tables: 5 and 2; Bi is showed in two tables: 3 and 6.Function ∂a1 ∂b1 ∂c1 ∂a2 ∂b2 ∂c2F1(χ, λi) xiRx2 yiRx2 Rx2 xiRy2 yiRy2 Ry2F2(χ, λi) xiGx2 yiGx2 Gx2 xiGy2 yiGy2 Gy2F3(χ, λi) xiBx2 yiBx2 Bx2 xiBy2 yiBy2 By2Table 1.Ai matrix for affine motion. First part.Function ∂α11 ∂α12 ∂α13 ∂α21 ∂α22 ∂α23 ∂α31 ∂α32 ∂α33 ∂β1 ∂β2 ∂β3F1(χ, λi) −R1 −G1 −B1 0 0 0 0 0 0 -1 0 0F2(χ, λi) 0 0 0 −R1 −G1 −B1 0 0 0 0 -1 0F3(χ, λi) 0 0 0 0 0 0 −R1 −G1 −B1 0 0 -1Table 2.Ai matrix for affine and projective motion. Second part.Function ∂R1 ∂G1 ∂B1F1(χ, λi) −α11 −α12 −α13F2(χ, λi) −α21 −α22 −α23F3(χ, λi) −α31 −α32 −α33Table 3.Bi matrix for affine and projective motion. First partIn the Tables, Rx1 , Ry1 , Rx2 , Ry2 , Gx1 , Gy1 , Gx2 , Gy2 , Bx1 , By1 , Bx2 and By2 have beenintroduced to simplify notation as follows: Rx1(qi), Ry1(qi), Rx2(q′i), Ry2(q′i), Gx1(qi),Gy1(qi), Gx2(q′i), Gy2(q′i), Bx1 (qi), By1 (qi), Bx2 (q′i) and By2 (q′i), respectively, beingRx1(qi), Ry1(qi),Gx1(qi), Gy1(qi), Bx1 (qi), By1 (qi), the components of the gradient oftheR,G,B bands, respectively, of test image at point qi; andRx2(q′i),Ry2(q′i),Gx2(q′i),Gy2(q′i), Bx2 (q′i), By2 (q′i) the components of the gradient of the R,G,B bands, respec-tively, of the reference image at point q′i.In addition, Nd, N1, N2, N3, N4, N5 and N6 have also been introduced as follows:Function ∂x ∂yF1(χ, λi) (a1Rx2 + a2Ry2)− (α11Rx1 + α12Gx1 + α13Bx1 ) (b1Rx2 + b2Ry2)− (α11Ry1 + α12Gy1 + α13By1 )F2(χ, λi) (a1Gx2 + a2Gy2)− (α21Rx1 + α22Gx1 + α23Bx1 ) (b1Gx2 + b2Gy2)− (α21Ry1 + α22Gy1 + α23By1 )F3(χ, λi) (a1Bx2 + a2By2 )− (α31Rx1 + α32Gx1 + α33Bx1 ) (b1Bx2 + b2By2 )− (α31Ry1 + α32Gy1 + α33By1 )Table 4.Bi matrix for affine motion. Second partfunction ∂a1 ∂b1 ∂c1 ∂a2 ∂b2 ∂c2 ∂d ∂eF1(χ, λi) xRx2NdyRx2NdRx2NdxRy2NdyRy2NdRy2Nd−xiRx2N1−xiRy2N2N2d−yiRx2N1−yiRy2N2N2dF2(χ, λi) xGx2NdyGx2NdGx2NdGy2NdyGy2NdGy2Nd−xiGx2N1−xiGy2N2N2d−yiGx2N1−yiGy2N2N2dF3(χ, λi) xBx2NdyBx2NdBx2NdxBy2NdyBy2NdBy2Nd−xiBx2N1−xiBy2N2N2d−yiBx2N1−yiBy2N2N2dTable 5.Ai matrix for projective motion. First part.Nd = (dxi + eyi + 1)N1 = a1xi + b1yi + c1, N2 = a2xi + b2yi + c2N3 =a1Nd − dN1N2d, N4 =a2Nd − dN2N2dN5 =b1Nd − eN1N2d, N6 =b2Nd − eN2N2d(12)The estimation process is resumed at Algorithm 1. A Feature Step is used to ini-tialize the motion estimator (whenever the deformation between images is quite largewe need a good initial vector of motion parameters). It mainly consists of a SIFT-basedtechnique [9] to detect and describe interest points, where for each interest point be-longing to the first image a K-NN search strategy is performed to find the k-closestinterest points at the second image. Finally, for estimating the first approximation of themotion parameters a random sampling technique is used [13].Regarding the illumination parameters at χ̂(0), they have initially been set to:Ω = Iand Φ = (0, 0, 0)T .5 Experiments and ResultsIn order to test the accuracy of the proposed motion estimation technique, several ex-periments have been perfomed using a set of challeging images. A preview of somefunction ∂x ∂yF1χ, λi (N3Rx2 +N4Ry2)− (α11Rx1 + α12Gx1 + α13Bx1 ) (N5Rx2 +N6Ry2)− (α11Ry1 + α12Gy1 + α13By1 )F2χ, λi (N3Gx2 +N4Gy2)− (α21Rx1 + α22Gx1 + α23Bx1 ) (N5Gx2 +N6Gy2)− (α21Ry1 + α22Gy1 + α23By1 )F3χ, λi (N3Bx2 +N4By2 )− (α31Rx1 + α32Gx1 + α33Bx1 ) (N5Bx2 +N6By2 )− (α31Ry1 + α32Gy1 + α33By1 )Table 6.Bi matrix for proyective motion. Second partInput: Images I1 = (R1, G1, B1)T and I2 = (R2, G2, B2)TOutput: χ̂, the vector of estimated motion parameters.1: Calculate image gradients.2: j = 0.3: Set Ω0 = I, Φ0 = (0, 0, 0)T and g0 = FeatureStep(I1, I2).4: χ̂(0) = (g0,p0)T , with p0 = (ω11, . . . , ω33, φ1, . . . , φ3).5: repeat6: j = j + 1.7: Update matrices Ai, Bi and ei using χ̂(j − 1).8: Estimate ∆̂χ(j).9: χ̂(j) = χ̂(j − 1) + ∆̂χ(j).10: until |∆̂χ(j)| is small enough.11: χ̂ = χ̂(j).Algorithm 1: Generalized Least Squares motion estimation algorithmof them is shown in Figure 1. In all image pairs there exists a geometric tranforma-tion simultaneus to a photometric one. Theses images have been obtained from severalsources, including: Bartoli’s examples 3, Brainard’s examples 4, Simon Fraser Univer-sity Computational Vision Lab’s examples 5 and Oxford’s Visual Geometry Group’sexamples 6. Finally, the last four have been acquired by ourselves using a conventionaldigital camera and varying the ilumination conditions.The Dual Inverse Compositional technique [2] was used for comparison purposessince it is, as far as we known, the only other technique that estimates simultaneuoslythe motion and the illumination parameters in color images. For each image pair, first,the feature step is performed to obtain a good inital motion parameters. Then, bothalgorithms are executed, obtaining two output parameters χGLS and χDIC . With theestimated parameters, the reference image can be transformed (geometrically and pho-tometrically). Then if the parameters have been correctly estimated, the resulting im-ages (IGLS and IDIC) have to be very similar to the corresponding reference images.Figure 2 shows the results obtained with the proposed technique for Bartoli’s images.First row shows the test and the reference image. Second row shows the resulting im-age and the panoramic image created. Note how both, the motion and the illuminationparameters have been correctly estimated.Once the resulting images (IGLS and IDIC) have been created it is possible to applya merit figure to measure to quality of the registration. Four similarity measures havebeen used. They are the Normalized Correlation Coefficient (NCC), the IncrementSign Correlation coefficient (ISC [6]), the selective correlation Coefficient (SCC [7])and the Normalized Average of Absolute errors (NAAE) defined as:3 http://www.lasmea.univ-bpclermont.fr/Personnel/Adrien.Bartoli/Research/DirectImageRegistration/index.html4 http://color.psych.upenn.edu/brainard/5 http://www.cs.sfu.ca/˜colour/data/objects_under_different_lights/index.html6 http://www.robots.ox.ac.uk/˜vgg/research/affine/index.htmlFig. 1. Input imagesNAAE(aae) ={0 if aae > TH(TH − aae)/TH otherwise , (13)where aae is the average of the absolute errors and TH is a constant.The four measures produce values from 0 (low similarity) to 1 (high similarity).Figure 3 shows the average of the values obtained for all experiments. Note how theproposed estimation technique overcomes Bartoli’s one for all similarity measures.6 Conclusionsbla blaReferences1. A. Bartoli. Groupwise geometric and photometric direct image registration. In Proceedingsof the British Machine Vision Conference, 2006.2. A. Bartoli. Groupwise geometric and photometric direct image registration. IEEE Transac-tions on Pattern Analysis and Machine Intelligence, 30:2098–2108, 2008.3. G. D. Finlayson, Mark S. Drew, and Brian V. Funt. Color constancy: generalized diagonaltransforms suffice. Journal of the Optical Society of America, A, 11:3011–3019, 1994.4. G. D. Finlayson, S. D. Hordley, and R. Xu. Convex programming colour constancy with adiagonal-offset model. In IEEE ICIP, 2005.5. G. Healey and A. Jain. Retrieving multispectral satellite images using physics-based in-variant representations. IEEE Transactions on Pattern Analysis and Machine Intelligence,18:842–848, 1996.6. Shunichi Kaneko, Ichiro Murase, and Satoru Igarashi. Robust image registration by incre-ment sign correlation. Pattern Recognition, 35(10):2223–2234, 2002.(a) Test Image (b) Reference Image(c) Resulting image (d) Panoramic ImageFig. 2. Results7. Shunichi Kaneko, Yutaka Satoh, and Satoru Igarashi. Using selective correlation coefficientfor robust image registration. Pattern Recognition, 36(5):1165–1173, 2003.8. R. Lenz, L. V. Tran, and P. Meer. Moment based normalization of color images. In IEEE 3rdWorkshop on multimdeia signal processing, pages 103–108, 1998.9. D. G. Lowe. Distinctive image features from scale-invariant keypoints. International Journalof Computer Vision, 60(2):91–110, 2004.10. F. Mindru, T. Tuytelaars, L. V. Gool, and T. Moons. Moment invariants for recognition underchanging viewpoint and illumination. Computer Vision and Image Understanding, 94:3–27,2004.11. Rau´l Montoliu and Filiberto Pla. Generalized least squares-based parametric motion estima-tion. Computer Vision and Image Understanding, 113(7):790–801, July 2009.12. L. Shao and M. Brady. Invariant salient regions based image retrieval under viewpoint andillumination variations. J. Vis. Commun. Image R., 17:1256–1271, 2006.13. P. H. S. Torr and A. Zisserman. Mlesac: A new robust estimator with application to estimat-ing image geometry. Computer Vision and Image Understanding, 78:138–156, 2000.14. G. West and M. H. Brill. Necessary and sufficient conditions for von kries chromatic adap-tation to give color constancy. J. Math. Biol., 15:249–258, 1982.15. G. Wyszecki and W.S. Stiles. Color science. John Wiley & Sons, 1967.16. Z. Xingming and Z. Huangyuan. An illumination independent eye detection algorithm. InIEEE ICPR, 2006.17. B. Zitova and J. Flusser. Image registration methods: a survey. Image and Vision Computing,21:997–1000, 2003.Fig. 3. Registration results

Color image registration under illumination changes

Crossref

Color Image Registration under Illumination Changes

Color image registration under illumination changes

Abstract

Similar works

Full text

Available Versions

Repositori Institucional de la Universitat Jaume I

Crossref