ISBN : 978-1-4244-5653-6International audienceIn this paper, we describe an automatic method for segmentation of building façade images. First, individual façades are isolated from general city block images. This step is based on accumulation of directional color gradients, assuming that façade structures are aligned. Then sky region is detected based on segmentation approach and color marker extraction. Finally, the images are split in floors using directional color gradient accumulation, as well. Our approach introduces several morphological filters to augment the robustness to problems such as: textured balconies, some specular reflections of the bright windows and small obstacles in images. The experimental results show the performance of our approach

Hernandez, Jorge

Marcotegui, Beatriz

HAL-MINES ParisTech

Morphological Segmentation of Building Fac¸ade ImagesJorge Hernandez, Beatriz MarcoteguiTo cite this version:Jorge Hernandez, Beatriz Marcotegui. Morphological Segmentation of Building Fac¸ade Images.16th IEEE International Conference on Image Processing (ICIP), Nov 2009, Le Caire, Egypt.IEEE, pp.4029-4032, 2009, <10.1109/ICIP.2009.5413756>. <hal-00834428>HAL Id: hal-00834428https://hal-mines-paristech.archives-ouvertes.fr/hal-00834428Submitted on 15 Jun 2013HAL is a multi-disciplinary open accessarchive for the deposit and dissemination of sci-entific research documents, whether they are pub-lished or not. The documents may come fromteaching and research institutions in France orabroad, or from public or private research centers.L’archive ouverte pluridisciplinaire HAL, estdestine´e au de´poˆt et a` la diffusion de documentsscientifiques de niveau recherche, publie´s ou non,e´manant des e´tablissements d’enseignement et derecherche franc¸ais ou e´trangers, des laboratoirespublics ou prive´s.MORPHOLOGICAL SEGMENTATION OF BUILDING FAC¸ADE IMAGESJorge Herna´ndez and Beatriz MarcoteguiMines ParisTechCMM- Centre de morphologie mathe´matiqueMathe´matiques et Syste`mes35 rue St Honore´ 77305-Fontainebleau-Cedex, FranceABSTRACTIn this paper, we describe an automatic method for seg-mentation of building fac¸ade images. First, individualfac¸ades are isolated from general city block images. Thisstep is based on accumulation of directional color gradi-ents, assuming that fac¸ade structures are aligned. Thensky region is detected based on segmentation approachand color marker extraction. Finally, the images are splitin floors using directional color gradient accumulation, aswell. Our approach introduces several morphological fil-ters to augment the robustness to problems such as: tex-tured balconies, some specular reflections of the brightwindows and small obstacles in images. The experimen-tal results show the performance of our approach.Index Terms— Mathematical morphology, fac¸adesegmentation, urban modeling1. INTRODUCTION3D urban environment modeling becomes a fundamen-tal part in a growing number of geo-applications as e.gGoogle Earth, Microsoft Virtual Earth, IGN - Geopor-tail 3D, etc. The main approaches to model cities arebased on coarse modeling, for instance: polyhedral rep-resentation, main walls, roof planes and ground planes.Nevertheless, recent research frameworks try to analyzebuilding fac¸ades in real images. This analysis extractsand reconstructs windows, doors and ornaments to pro-vide rich information of the buildings and to add visualrealism. Furthermore, the reconstruction of fac¸ades isa hard problem, given the large variety in their appear-ances and structures [1, 2]. The building fac¸ade analysishas been widely studied in the literature. In general,the methods use rectified and cropped images. Lee andNevatia in [3] and Wang et al. [2] developed a methodto detect windows from street level images of isolatedbuildings. Mu¨ller et al. in [4] find repetitive architec-tural structures by using mutual information to describe{ hernandez,marcotegui }@cmm.ensmp.fra single fac¸ade image; where the fac¸ade is manually ex-tracted.However when we take a picture of city blocks fromthe street level, several problems appear to model a singlefac¸ade: 1- Variations of resolution caused by perspectiveeffects, 2- Multiple fac¸ades in images, 3- Sky information,4- Road and sidewalk information, 5 - Ground floor; theground floor is considered a problem due to non repeti-tive structures [4]. These problems are illustrated as aninput image of Fig. 1.In this paper we present an automatic methodthat extracts isolated fac¸ades from city block images.This segmentation facilitates the extraction of seman-tic/grammatical information from a single ortho-fac¸adeand an efficient grammatical modeling of fac¸ades be-comes possible.This paper is organized as follows. Section 2 describesour method. Vertical splitting in isolated fac¸ades is pre-sented in Section 2.1. Sky detection and horizontal split-ting are shown in Section 2.2 and Section 2.3, respec-tively. Section 3, experimental results are shown and theperformance of our method is illustrated. Finally, con-clusions are drawn in Section 4.2. SEGMENTATION OF BUILDINGFAC¸ADESA block diagram and an example of the method are givenin Fig. 1. The first step of our method is image recti-fication. We have developed an automatic rectificationalgorithm by using vanishing point detection via Houghtransform [5].2.1. Vertical SplittingWe assume that fac¸ade structures (windows, doors, bal-conies) are horizontally and vertically aligned [3]. Forthat reason, our approach is based on gradient projectionanalysis. For vertical splitting only horizontal gradientis used. The horizontal color gradient (Gh) is generatedby using a vertical structuring element (SE). With thisFig. 1. Segmentation of building fac¸ade method. Input image: picture of city blocks from street level and Output required image formodelling purposes.assumption, the approach is more robust to generatedgradients by noise or spurious structures in other direc-tions. Fig. 2(a) shows (Gh). Afterwards, we accumulateon columns (Gh), providing vertical projection Pv (Gh).Nevertheless, in textured fac¸ades, the gradient accu-mulation Pv (Gh) presents two important problems: thecreation of false peaks and false valleys and the reductionof dynamic between them. These problems produce anerroneous fac¸ade division. To solve this problem a filter-ing operation is included, before the gradient projection.We apply a directional morphological opening of λ size.The directional opening is applied in the same directionof gradient, i.e horizontal opening for Gh image. Thesize opening value is estimated using the smallest possi-ble window size. The filtered gradient (G˜h) is shown inFig. 2(b).(a) Gh and Pv (Gh) (b) G˜h and Pv(G˜h)(c) Fac¸ade divisionFig. 2. (a) Horizontal color gradient and projection, (b) Filteredgradient and projection. (c) Vertical fac¸ade division.Pv(G˜h)profile contains peaks at windows locationsand valleys within the wall. The profile is filtered byusing an alternate sequential filter (ASF ) to allow aneasy detection of the most representative maxima andminima of Pv(G˜h). Finally, to find the fac¸ade divi-sion we use watershed (WS) segmentation of invertedASF(Pv(G˜h)). The cost of this segmentation is verylow thanks to the fact that operation is applied to pro-files (1D images). Vertical division result is illustrated inFig. 2(c).Among found divisions, we have two division types:intra - fac¸ades and inter - fac¸ades. We can see onPv(G˜h)profile, inter - fac¸ade divisions have lower valu-ation than intra - fac¸ade divisions. This is due to the factthat there are not horizontal structures in the separationline between two buildings, while intra-fac¸ade divisionscross balcony handrails, ornaments or floor separations,leading to a higher profile valuation. Hence, to classifyboth division types a k-means clustering algorithm isused. We assume as initial cluster seeds, absolute min-imum and maximum data value. At last, the divisionsbelonging to minimum seed cluster are the inter - fac¸adeseparations (Fig. 2(c): red lines).2.2. Sky EliminationAs a result of perspective correction, several black zonesappear in the image. These zones produce some prob-lems in sky detection and horizontal splitting. Thus, asimple method to remove top and bottom black zones isimplemented.Sky detection is a common problem in pattern recog-nition where the number of patents has been very signif-icant. Sky information has been used to classify indoor-outdoors scenes [6], to know the orientation of images[7]and as a marker to detect the semantic of urban images[2]. We propose a method, applied to urban images,based on connected component extraction and color in-formation. We combine two approaches as a startingpoint: combination of a segmentation of connected com-ponents and a color marker [8]; and seed region detectionto characterize the sky color [7].Fig. 3. Sky de-tection process dia-gram.Our method has the followingsteps: First, the image is seg-mented by using ultimate open-ing (UO) operator [9]. In paral-lel, a marker of blue color is cal-culated. Hence, in the third stepwe extract candidate regions of thesky, based on color and positioncriteria. Next, a seed sky regionis selected and characterized by aGaussian Model. Finally, all candi-dates regions whose color compati-ble with the computed color modelare added to the sky result. Thesesteps are translated into Fig. 3. The method is a rel-atively simple to detect the sky. More structured ap-proaches [6, 7] are available to improve the robustness.Moreover, these methods look for all CCs of the imageto select candidate regions. In our case, this is not thebest solution because of window reflection problem in ur-ban images which produces a lot of false positives. Fig.4 shows some examples of the experimental results bydemonstrating the performance of our method. We il-lustrate characteristic cases of the sky in urban imageswhere our method has correctly detected it.Subsequently to sky detection, the sky is eliminatedhorizontally and vertically, if it is necessary. Horizon-tally, we cut sky off, where horizontal sky edges have thelargest number of pixels; after that a similar procedureis applied on vertical direction.Fig. 4. Sky detection and lines of cropping images.2.3. Horizontal SplittingBy using the same method of gradient projection pre-sented in Section 2.1, a first horizontal division is carriedout. In this case a vertical color gradient (Gv) is pro-duced by a horizontal SE (Fig. 5(a)). Then, gradientimage is filtered by a vertical opening by generating G˜v(Fig. 5(b)). After that, we project G˜v on its perpen-dicular direction to create Ph(G˜v)profile (Fig. 5(c)).Next, the profile is filtered by: an ASF as in verticalsplitting case, and a h-Minima operator [10]. h-Minimafilter eliminates minima which have a depth lower than athreshold h. This operator is added to avoid divisions onhomogeneous regions such as roads. Lastly, we use WSsegmentation of inverted filtered profile to find horizontaldivisions.(a) Gv (b) G˜v (c) Ph(G˜v)(d) Ph(G˜h)Fig. 5. (a) Horizontal color gradient. (b) Filtered gradient.(c) Horizontal projection of G˜v and horizontal fac¸ade division.(d)Horizontal projection of G˜h and horizontal fac¸ade division re-fining.We have found horizontal divisions thanks to windowgradients (Fig. 5(b)). For that reason, floor separationsare not precisely located. To solve this problem, we movethe first divisions to the highest horizontal gradient (G˜h)for each floor, given as a result Fig. 5(d).In order to eliminate the ground floor and road sec-tion, we remove the last section iteratively. In thisexample, two sections should be removed. To deter-mine if a division will be eliminated, we compute thedifference between normalized Pv(G˜h)and normalizedPv(G˜h(floor))for each floor.Fig. 6. Analysis of sections.Note that the difference of a normal floor is lowerthan difference of ground floor (Fig. 6). This differencebetween dynamics is due to the gradient produced bywindows. If the section difference is bigger than differ-ence mean, the section is eliminated.3. EXPERIMENTAL RESULTSThe presented method was tested on images acquired byuncalibrated cameras. The images correspond to approx-imately ten urban blocks of the 5th Paris district. More-over, we have tested the same technique on problematicimages (Fig. 7) by showing that the proposed methodis robust to problems such as: textured balconies, somespecularity on the windows, small obstacles (trees, flow-erpots). The method presents good results on the wholedataset (all databases and result tests can be found onthe following web site1).(a) (b) (c)(d) (e) (f)Fig. 7. Images with several obstacles (a-c) Input images (d-f)Output images4. CONCLUSIONSThe major contribution of this paper is a novel methodfor morphological segmentation of building fac¸ade im-ages. From an urban block picture taken from theground level, we obtain a cropped ortho-fac¸ade image.The method is completely automatic with the same pa-rameters for the whole dataset. Several morphologicalfilters are added through the process, by increasing therobustness of method. The vertical and horizontal divi-sion is based on profile analysis. The profiles are createdby projection of directional color gradients. Moreover,we have developed a sky detection method from urbanimages. The method has been designed to detect daylight skies via connected component segmentation andcolor marker extraction. The results obtained are sat-isfactory on the dataset. Extended tests on a larger1http://cmm.ensmp.fr/∼hernandez/results/FacSegFromUrbanBlocks/indexFacSegFromBlocks.htmdatabase are foreseen in the framework of the TerraNu-merica project.5. ACKNOWLEDGESThe work reported in this paper has been performedas part of Cap Digital Business Cluster TerraNumericaproject.6. REFERENCES[1] Luc Van Gool, Gang Zeng, Filip Van den Borre,and Pascal Mu¨ller, “Towards mass-produced build-ing models,” in PIA’07: Photogrammetric ImageAnalysis, September 2007, pp. 209–220.[2] X. Wang, S. Totaro, F. Taillandier, A. Hanson,and S. Teller, “Recovering fac¸ade texture and mi-crostructure from real-world images,” in ECCV Tex-ture 2002 Workshop, Copenhagen, Denmark, 2002.[3] Sung Chun Lee and Ram Nevatia, “Extraction andintegration of window in a 3d building model fromground view images,” IEEE Computer Vision andPattern Recognition, vol. 02, pp. 113–120, 2004.[4] Pascal Mu¨ller, Gang Zeng, Peter Wonka, andLuc Van Gool, “Image-based procedural modelingof facades,” ACM Transactions on Graphics, vol.26, no. 3, pp. 85–93, 2007.[5] R. I. Hartley and A. Zisserman, Multiple View Ge-ometry in Computer Vision, Cambridge UniversityPress, ISBN: 0521540518, second edition, 2004.[6] Jiebo Luo and Stephen P. Etz, “A physical model-based approach to detecting sky in photographic im-ages,” IEEE Transactions on Image Processing, vol.11, no. 3, pp. 201–212, 2002.[7] Andrew C. Gallagher, Jiebo Luo, and Wei Hao, “Im-proved blue sky detection using polynomial modelfit,” in IEEE International Conference on ImageProcessing, ICIP’04, 2004, pp. 2367–2370.[8] Valery Risson, Application de la Morphologie Math-e´matique a` l’Analyse des Conditions d’e´clairage desImages Couleur, Ph.D. thesis, CMM - Mines Paris-Tech, 2001.[9] Serge Beucher, “Numerical residues,” Image andVision Computing, vol. 25, no. 4, pp. 405–415, 2007.[10] Pierre Soille, Morphological Image Analysis: Prin-ciples and Applications, Springer-Verlag New York,Inc., Secaucus, NJ, USA, 2003.

Morphological Segmentation of Building Façade Images

HAL Descartes

Crossref

Morphological segmentation of building façade images

https://hal-mines-paristech.archives-ouvertes.fr/hal-00834428/file/Hernandez_Marcotegui_icip09.pdf

Morphological Segmentation of Building Façade Images

Abstract

Similar works

Full text

Available Versions

HAL-MINES ParisTech

HAL Descartes

Crossref