In this paper, we present a new algorithm for color-based object recognition that detects objects and estimates their pose (position and orientation) in cluttered scenes observed under uncontrolled illumination conditions. As with so many other color-based object-recognition algorithms, color histograms are also fundamental to our approach; however, we use histograms obtained from overlapping subwindows, rather than the entire image. Furthermore, each local histogram is normalized using greyworld normalization in order to be as less sensitive to illumination as possible. An object from a database of prototype objects is identified and located in an input image by matching the subwindow contents. The prototype is detected in the input whenever many good histogram matches are found between the subwindows of the input image and those of the prototype. In essence, normalized color histograms of subwindows are the local features being matched. Once an object has been recognized, its 2D pose is found by approximating the geometrical transformation most consistently mapping the locations of prototype’s subwindows to their matched subwindow locations in the input image

Funt, B.

Macaire, L.

Muselet, D.

Shi, L.

Simon Fraser University Institutional Repository

OBJECT RECOGNITION AND POSE ESTIMATION ACROSSILLUMINATION CHANGESMuselet D.Laboratoire LIGIV EA 3070 - Universite´ Jean Monnet - Francedamien.muselet@univ-st-etienne.frFunt B., Shi L.School of Computing Science, Simon Fraser University, Vancouver, Canadafunt@cs.sfu.ca, lshia@cs.sfu.caMacaire L.Laboratoire LAGIS UMR CNRS 8146 - Universite´ des Sciences et Technologies de Lille - Franceludovic.macaire@univ-lille1.frKeywords: Color histograms, object recognition, 2D pose estimation, illumination changes, local color descriptorsAbstract: In this paper, we present a new algorithm for color-based object recognition that detects objects and estimatestheir pose (position and orientation) in cluttered scenes observed under uncontrolled illumination conditions.As with so many other color-based object-recognition algorithms, color histograms are also fundamental toour approach; however, we use histograms obtained from overlapping subwindows, rather than the entireimage. Furthermore, each local histogram is normalized using greyworld normalization in order to be as lesssensitive to illumination as possible. An object from a database of prototype objects is identified and locatedin an input image by matching the subwindow contents. The prototype is detected in the input whenever manygood histogram matches are found between the subwindows of the input image and those of the prototype.In essence, normalized color histograms of subwindows are the local features being matched. Once an objecthas been recognized, its 2D pose is found by approximating the geometrical transformation most consistentlymapping the locations of prototype’s subwindows to their matched subwindow locations in the input image.1 INTRODUCTIONStarting with Swain and Ballard’s color index-ing (Swain and Ballard, 1991), color has proved tobe a very important clue for object recognition. Fol-lowing in this tradition, we present a new algorithmfor color-based object recognition that detects objectsand estimates their pose (position and orientation) incluttered scenes under uncontrolled illumination con-ditions. As with so many other color-based object-recognition algorithms (Funt and Finlayson, 1995;Bressan et al., 2003), color histograms are also funda-mental to our approach; however, we use histogramsobtained from overlapping subwindows, rather thanthe entire image. Furthermore, each local histogramis normalized using greyworld normalization (Buchs-baum, 1980).An object from a database of prototypeobjects is identified and located in an input image bymatching the subwindow contents. The prototype isdetected in the input whenever many good histogrammatches are found between the subwindows of theinput image and those of the prototype. In essence,normalized color histograms of subwindows are thelocal features being matched. Once an object hasbeen recognized, its 2D pose is found by approximat-ing the geometrical transformation most consistentlymapping the locations of prototype’s subwindows totheir matching subwindow locations in the input im-age (Lowe, 1999).An entry in the database of prototypes is built froman image of a single object placed on an uniformbackground. The test images containing the objectsto be recognized may contain several objects, some ofwhich may be partially occluded. The prototype andtest images are acquired under different illuminationconditions and with the same zoom parameters (SeeFig.1).Color histograms are very effective for objectrecognition (Swain and Ballard, 1991) and image in-dexing (Park et al., 1999) because they are simple andfast to compute, are invariant to rotation and trans-lation, and are insensitive to partial object occlusion.However, color histograms of whole objects are use-less for determining object pose, precisely becausethey are rotation invariant. In terms of accurately de-termining object position, Swain’s histogram back-Example prototype images.Example input test images.Figure 1: These images come from the Simon FraserUniversity Database (Barnard et al., 2002) available athttp://www.cs.sfu.ca/∼colour/data.projection (Swain and Ballard, 1991) is very sensi-tive to noise and provide only a coarse estimate ofthe object’s position. When the input images maycontain several objects and include the possibilityof partial occlusion, many other non-color-based ap-proaches (Lowe, 1999; Ohba and Ikeuchi, 1997) relyon local image features for matching. The approachproposed here combines the ideas of color histogrammatching and local feature matching.Many of the local-feature-based object recogni-tion methods (Bressan et al., 2003; Lowe, 1999; Ohbaand Ikeuchi, 1997) extract interest points from theimages as an initial step and then evaluate local de-scriptors around these points. A drawback of this ap-proach is that the robustness and the repeatability ofthe interest-point detector becomes crucial. To avoidthe reliability problems associated with interest-pointdetectors, we propose to analyze all the local neigh-borhoods in the image and to extract descriptors forall of them. Indeed, rather than to extract featuresfrom a limited number of areas in the image, we con-sider all pixels to be interest points and consequentlyextract the features around all of them. Thus wedivide the image into overlapping subwindows andcompute their color histograms. The computation inthis step can be organized so that each pixel needs tobe visited only once, so it is fast.When the illumination is not controlled duringthe acquisition of the images, the classical color his-tograms lead to poor recognition results (Funt et al.,1998). Thus, we propose to normalize the color his-tograms in order to cope with this problem. One clas-sical and computationally simple approach is grey-world normalization. The main drawback of this nor-malization is that it assumes that the illumination isspatially constant over the whole image (Buchsbaum,1980). In normalizing each subwindow separately, weonly assume that the illumination is constant withineach subwindow, not across the whole image.The database of prototypes represents each objectin terms of the normalized local color (NLC) his-tograms from its image’s subwindows. This repre-sentation associates a point on the object with eachNLC histogram. To identify the objects in the inputtest image, each test-image subwindow is matched tothe entire set of database subwindows and labeled ac-cording to the one that matches the best. A subsetof subwindows with the same object label indicatesthe presence of the corresponding object in the im-age. The locations of the subwindow labels withinthe image indicates the object’s pose.The second section of this paper is about the il-lumination changes and the greyworld normalization.The third section presents details about how the spaceand time requirements for NLC histogram storage andmatching can be reduced using incremental principalcomponents analysis (Hall et al., 1999). The specificsof 2D-pose estimation are described in the fourth sec-tion. Results of tests based on the Amsterdam imagedatabase (Geusebroek et al., 2005) are given in thefifth section.2 ILLUMINATION CHANGESIn order to deal with variations in the spectral com-position of the incident illumination, we adopt the di-agonal model of illumination change. The diagonalmodel assumes that the spectral sensitivity functionof each sensor of the camera is sufficiently narrow-band that they can be viewed as Dirac delta functionsat three distinct wavelengths. In practice, althoughthis assumption does not hold perfectly it is generallyan adequate model, and it can be improved by spectralsharpening (Finlayson et al., 1994).Using the diagonal model of illumination changealong with the additional assumption that all pixelsin a subwindow are lit by the same illumination, wecan apply the greyworld normalization to each localcolor histogram by dividing each color component byits mean value within this subwindow. Each subwin-dow within then is characterized by a normalized lo-cal color (NLC) histogram.3 EIGEN NORMALIZED LOCALCOLOR HISTOGRAMSSince each prototype image represents only one ob-ject, each subwindow represents a specific area of theobject. Thus, considering that we have P prototypeimages Iipro, i ∈ {1, ...,P}, the prototype image Iiprowhich represents the object Oi, is divided into WPisubwindows wpij, j ∈ {1, ...,WPi}, each one repre-senting the jth area Opij of the object Oi.Since the proposed object recognition method re-quires the storage and matching of many subwindowhistograms, it is important to reduce the memory andcomputation requirements as much as possible. Onestrategy for decreasing the complexity of histogrammatching is to reduce the dimensionality of the his-tograms (Bressan et al., 2003; Tran and Lenz, 2005).Therefore, we apply principal component analysis tothe set of prototype local color histograms.Following the method of Tran and Lenz (Tran andLenz, 2005), PCA is applied to histogram differences,rather than the histograms themselves. The histogramdifferences suffice since the aim when compressinghistograms for the object recognition is not to be ableto reconstruct the histograms, but only to estimate dis-tances between histograms. Therefore, PCA appliedon the space of histogram differences should leadto better results than PCA applied on the histogramspace. Since we care only about similar images, PCAis not applied on all the differences between the proto-type histograms, but only on the differences betweensimilar prototype histograms. Hence, for each proto-type histogram, we use the histogram difference be-tween it and its closest prototype histogram from thesame image. The closest histogram is the one at theminimum Manhattan distance between the two his-tograms. Swain showed that the Manhattan distanceis equivalent to use of the intersection between colorhistograms when these histograms contain the samenumber of pixels (Swain and Ballard, 1991).When the number of images in the prototypedatabase is high, the number of NLC histograms be-comes very high and the time required to apply prin-cipal component analysis becomes prohibitive. Toovercome this limitation, we move to incrementalPCA (Hall et al., 1999). Thanks to incremental PCA,the size of the prototype database is effectively unlim-ited. The IPCA step is completed once off-line.The NLC histograms projected onto the eigenba-sis from IPCA are then called eigen NLC (ENLC)histograms. All NLC histograms, from both thedatabase of prototypes and the input test image, areprojected onto the same eigenbasis. Finally, eachinput ENLC histogram is compared against all theprototype ENLC histograms, and the most similarprototype ENLC histogram is kept. Histograms arematched according to the Manhattan distance be-tween them.After the matching step, each sub-windowwqk, k ∈ {1, ...,WQ}, of the input imageis associated with one prototype subwin-dow wpij, and so, with one object area Opijof the object Oi. The subwindow’s labels[input subwindow, ob ject area] [wqk,Opij] areused to determine the best geometrical transforma-tion mapping the corresponding prototype image tothe input image.4 2D POSE ESTIMATIONAfter the matching step, the subwindows wqk,k ∈ {1, ...,WQ}, from the input image will havean associated object area Opij. Let Ci, i ∈{1, ...,T}, T ≤ WPi, denote the subset of fea-tures (areas) from the object Oi that have been as-sociated with at least one input subwindow wqk:Ci = {Opij|there exists k so that [wqk,Opij] exists}.We next consider the non-empty subsets Ci one by oneand estimate the orientation and position of the corre-sponding object Oi in the input image. This meansfinding the geometric transformation from the spatialcoordinates of the object Oi in the prototype imageIipro to its coordinates in the input image. The esti-mation of this transformation is based on the spatialcoordinates of the matching subwindow pairs.As described by Lowe (Lowe, 1999), the geomet-ric transformation from a point [x,y]T associated witha prototype subwindow to a point [u,v]T associatedwith the corresponding input subwindow can be writ-ten as:[x −y 1 0y x 0 1]m1m2txyy=[uv](1)where tx and ty represent the translation parametersand the mi represent the rotation around the center ofthe object and scale parameters.This equation is based on one pair of prototype and in-put subwindows, but we can add some other pairs andcalculate the least-squares solution for the geometricparameters.Since this method is very sensitive to outliers, wepropose the following two step approach:• The set Ci is randomly divided into subsetsof a fixed number of features, and then theleast-squares fit for the geometric transformationfor each of these subsets is determined indepen-dently. A large residual error in the fit indicatesmismatched features that are then deleted fromthe set Ci. The number of features in a subsetand the threshold of the residual error are fixedparameters determined experimentally.• Using only those features leading to low resid-ual error in the preceding step, the best geomet-ric transformation is determined by least-squaresfitting.5 EXPERIMENTAL RESULTSWe first test our algorithm on the real images fromFig.1 and the recognition and pose estimation are per-fect. Then, the Amsterdam Library of Object Images(ALOI) database (Geusebroek et al., 2005) is used fortesting. The Amsterdam database contains 12 sets ofcolor images. Each set contains images of one ob-ject on a uniform background under one of the 12 dif-ferent illuminants having color temperatures between2175◦K to 3075◦K. For the tests, we use 2 sets ofcolor temperature 2325◦K and 2750◦K. 250 imagesof the first set are used as the prototype images. Fromthe second set, we extract 100 objects to create 20input images, each one representing 5 objects. Eachobject is subject to 2D rotation and translation beforebeing added to the set of input images.For these tests, the size of the subwindows is fixedat 45x45 pixels, and the offset between the centers oftwo neighboring subwindows is 15 pixels. The aver-age number of ENLC histograms for each prototypeimage is 250. The number of bins in a raw histogramis 83 = 512. After projection on the eigenbasis, thisnumber reduces to 64.The algorithm correctly recognizes and makes aperfect estimate of the pose for 96 of the 100 inputobjects.6 CONCLUSIONA method for object recognition and 2D pose es-timation has been presented. The method is insensi-tive to the color of the scene illumination. The ba-sic strategy is to match local image features, in par-ticular, to match the color histograms of subwindowsfrom the input image to histograms of subwindows ofprototypes in the database. The subwindow contentsare normalized via greyworld averaging to remove theeffects of variations in illumination. Pose is deter-mined by finding the best correspondences betweenthe matching subwindows that are consistent with asingle geometrical transformation. Overall the accu-racy of the proposed method is quite good consider-ing that the database comprises images of objects withquite similar color distributions imaged under lightsof different color temperature than the input images.REFERENCESBarnard, K., Martin, L., Funt, B., and Coath, A. (2002).A data set for colour research. Color Research andApplication, 27(3):147–151.Bressan, M., Guillamet, D., and Vitria, J. (2003). Using anica representation of local color histograms for objectrecognition. Pattern Recognition Letters, 36:691–701.Buchsbaum, G. (1980). A spatial processor model for ob-ject colour perception. Jour. of the Franklin Institute,310:1–26.Finlayson, G., Drew, M., and Funt, B. (1994). Color con-stancy: Generalized diagonal transforms suffice. Jour.of the Optical Society of America A, 11(11):3011–3020.Funt, B., Barnard, K., and Martin, L. (1998). Is machinecolour constancy good enough? In Procs.of the 5thEuropean Conf. on Computer Vision, pages 445–459.Funt, B. and Finlayson, G. (1995). Color constant color in-dexing. IEEE Trans. on Pattern Analysis and MachineIntelligence, 17(5):522–529.Geusebroek, J. M., Burghouts, G. J., and Smeulders, A.W. M. (2005). The amsterdam library of object im-ages. Int. Jour. of Computer Vision, 61(1):103–112.Hall, P., Marshall, D., and Martin, R. (1999). Adding andsubtracting eigenspaces. British Machine Vision Con-ference, 2:463–472.Lowe, D. G. (1999). Object recognition from local scale-invariant features. In Procs. of the International Conf.on Computer Vision, pages 1150–1157, Corfou.Ohba, K. and Ikeuchi, K. (1997). Detectability, uniqueness,and reliability of eigen windows for stable verificationof partially occluded object. IEEE Trans. on PatternAnalysis and Machine Intelligence, 19 (9):1043–1048.Park, D., Park, J., Kim, T., and Han, J. (1999). Image in-dexing using weighted color histogram. In Procs. ofICIAP, pages 909–914.Swain, M. J. and Ballard, D. H. (1991). Color indexing. Int.Jour. of Computer Vision, 7(1):11–32.Tran, L. V. and Lenz, R. (2005). Compact colour descrip-tors for colour-based image retrieval. Signal Process.,85(2):233–246.

Object Recognition and Pose Estimation across Illumination Changes

http://summit.sfu.ca/system/files/iritems1/18255/Muselet_Funt_Shi_Macaire_VISAPP_2007.pdf

Object Recognition and Pose Estimation across Illumination Changes

Abstract

Similar works

Full text

Available Versions

Simon Fraser University Institutional Repository