A huge archive of very high-resolution SAR and optical satellite imagery acquired by different remote sensing satellites provides the opportunity to explore the possibility of 3D-reconstruction by multi-sensor stereogrammetry. This paper investigates the potential of SAR-optical stereogrammetry over urban areas using very-high-resolution imagery acquired by TerraSAR-X and Worldview-2. Furthermore, the potentials and challenges of deriving simple prismatic building models by combining the stereogrammetry results and OpenStreetMap building footprints are discussed. The results of this data fusion research demonstrate the possibility of using SAR-optical stereogrammetry for urban 3D reconstruction at level-of-detail 1

Bagheri, Hossein

Schmitt, Michael

Zhu, Xiao Xiang

Crossref

Towards the Reconstruction of Prismatic Building Models by SAR-Optical Stereogrammetry

Institute of Transport Research:Publications

Towards the Reconstruction of Prismatic BuildingModels by SAR-Optical StereogrammetryHossein Bagheri1, Michael Schmitt1, Xiao Xiang Zhu1,21Signal Processing in Earth Observation, Technical University of Munich (TUM)Arcisstr. 21, 80333 Munich, Germany2Remote Sensing Technology Institute (IMF), German Aerospace Center (DLR)Mu¨nchener Str. 20, 82234 Wessling, GermanyAbstract—A huge archive of very high-resolution SAR andoptical satellite imagery acquired by different remote sensingsatellites provides the opportunity to explore the possibility of3D-reconstruction by multi-sensor stereogrammetry. This pa-per investigates the potential of SAR-optical stereogrammetryover urban areas using very-high-resolution imagery acquiredby TerraSAR-X and Worldview-2. Furthermore, the potentialsand challenges of deriving simple prismatic building modelsby combining the stereogrammetry results and OpenStreetMapbuilding footprints are discussed. The results of this data fusionresearch demonstrate the possibility of using SAR-optical stere-ogrammetry for urban 3D reconstruction at level-of-detail 1.Index Terms—data fusion, stereogrammetry, 3D reconstruc-tion, dense matching, urban areas, city modelsI. INTRODUCTIONOne particular interest in remote sensing is the 3D recon-struction of urban areas for diverse applications such as 3Dcity modelling, urban and crisis management etc. A typicalmethod for reconstructing urban areas on a large scale is toemploy stereo optical imagery provided by high-resolutionspace-borne sensors in an ideal acquisition situation. However,because of limitations in acquiring those ideal images such ascloud effects as well as limited absolute localization accuracy,optical stereo might not always be the optimal choice. Incontrast, the amplitude images provided by SAR sensors donot suffer from the aforementioned issues, and can thus pro-vide input to time-critical 3D reconstruction tasks. Regardingthe growing archive of very high-resolution SAR and opticalimagery, developing a framework that takes advantage of bothSAR and optical imagery can provide a great opportunityto produce 3D spatial information over urban areas as anapplication of data fusion in remote sensing [1].In the literature, a few papers can be found that dealtwith the combination of SAR and optical imagery for the3D reconstruction of urban objects, e.g. [2]. However, noneof them has exploited multi-sensor stereogrammetry for thispurpose yet. In this research, we focus on the potential of3D reconstruction from very high-resolution SAR-optical im-age pairs such as TerraSAR-X/WorldView-2 through a densematching process. To the stereogrammetric processing chain,a simple approach for the generation of 3D prismatic buildingmodels by combining the derived height information withOpenStreetMap building footprints is added.II. RECONSTRUCTING BUILDING MODELS FROMSAR-OPTICAL STEREOGRAMMETRYA full framework for stereogrammetric 3D reconstructionfrom SAR-optical image pairs was presented in [3], [4]. Itconsists of several steps: generating rational polynomial coef-ficients (RPCs) for each image to replace the different physicalimaging models by a homogenized mathematical model; RPC-based multi-sensor block adjustment to enhance the relativeorientation between both images; establishing a multi-sensorepipolarity constraint to reduce the matching search spacefrom 2D to 1D.The main step in SAR-optical stereogrammetry, however, isto estimate disparity values by using a dense matching algo-rithm such as semi-global matching (SGM). In the following,more details of SAR-optical dense matching along with exper-imental results of TerraSAR-X/WorldView-2 stereogrammetryover urban areas will be presented. After that, the potentialsand challenges of using the thus-produced 3D point clouds forthe generation of simple prismatic models are discussed.A. SAR-Optical Dense MatchingThe core challenge in SAR-optical stereogrammetry is tofind disparity maps between two images by using a densematching algorithm. For the presented research, we haveinvestigated the application of classical SGM for that purpose.SGM computes the optimum disparity maps by minimizingan energy functional which is constructed by a data and afidelity term [5]. While the data term is defined by a similaritymeasure, the fidelity term employs two penalties to smooththe final disparity map. Because of aggregating cost valuescomputed by a cost function in the heart of SGM alongwith a regularizing smoothness term, SGM is relatively robustagainst noise, which seems to make it a good choice forSAR-optical stereogrammetry. However, some specific settingsmust be considered to adapt SGM to SAR-optical image pairs:One of the main impact factors on the performance of SGMfor dense matching of SAR-optical imagery is the robustnessand strength of cost functions used in SGM to measurethe similarity between SAR and optical images because ofthe complicated radiometric relationship. According to [6],pixel-wise Mutual information (MI), and Census are moreappropriate for difficult illumination relationships than, e.g.978-1-7281-0009-8/19/$31.00 ©2019 IEEEnormalized cross-correlation (NCC). In this research, we haveinvestigated all conventionally used similarity measures, i.e.pixel-wise MI, Census, their weighted sum, and NCC.Next, the minimum and maximum disparity values are usedto limit the length of the search domain along epipolar curves.Those values can be achieved from an external DEM likeSRTM or selected according to the mean height of the studyarea, considering a height threshold suited for the target urbanarea.The 3D-reconstruction results achieved with these SGMsettings are shown in Section III.B. Derivation of Prismatic Building ModelsOne of the most interesting applications of modern pho-togrammetric 3D reconstruction is to produce 3D buildingmodels [7]. There are different levels of modeling whichhave been gathered under the standard of the City GeographyMarkup Language (CityGML). These are summarized in [8].Fig. 1: Different Levels of building models in CityGML [9]Figure 1 displays different levels-of-detail as defined inthe CityGML standard. As shown in this figure, the lowestlevel of detail (LOD) is 1 (LOD1), which simply modelsbuildings as boxes without considering any details. Thus,for LOD1 only the outlines of buildings along with heightinformation are required. Regarding the output of SAR-opticalstereogrammetry being a relatively noisy and sparse pointcloud, an inclusion of auxiliary data is advised: We thus choseto extract the building footprints from the OpenStreetMapbuilding layer, while the height information can be derivedfrom the output of SAR-optical stereogrammetry. This is doneby assigning the median height of all points found withina building outline to the footprint and thus extruding thefootprint into the height axis as well.III. EXPERIMENTS AND RESULTSA. Test DataFor the experiments presented in this paper, a high-resolution TerraSAR-X/WorldView-2 image pair, acquiredover the city of Berlin, Germany, is used. For the pre-processing, first, the SAR image was filtered by a non-localfilter to reduce the speckle. After that, they were resampledto 1m × 1m pixel size to homogenize the study sceneswith respect to better similarity estimation. After multi-sensorbundle adjustment, sub-images from the overlapped part of thestudy area were selected. These sub-images are displayed inFig. 2. The specifications of the TerraSAR-X and WorldView-2 images are provided in Tab I.Fig. 2: Display of study subsets located in Berlin, left:WorldView-2, right:TerraSAR-XFig. 3: Achieved point cloud from stereogrammetric 3D recon-struction of TerraSAR-X/WorldView-2 over the Berlin studyareaB. Dense Matching ResultsBy using SGM for dense matching of the TerraSAR-Xand WorldView-2 image pair, a sparse point cloud could beproduced. As mentioned in Section II-B, we tried differentsimilarity measures in the SGM data term. As an example,the achieved point cloud by using MI as a similarity measureis shown in Fig. 3.To validate the accuracy of the resulting 3D point clouds,we employed an accurate LiDAR point cloud with an averagedensity of 6 points per square meters and a vertical accuracyof 20cm. For accuracy calculation, after Least Square (LS)plane fitting on k (here: k = 6 points) nearest neighbors ofTABLE I: Specifications of the TerraSAR-X and WorldView-2 imagesSensor Acquisition Mode Off-Nadir Angle (◦) Ground Pixel Spacing (m) Acquisition dateTerraSAR-X Spotlight 36.11 0.17 × 0.45 04.2016WorldView-2 Panchromatic 29.1 0.5 × 0.5 05.2013each target point in the reference point cloud, the Euclideandistance between the target point to the fitted reference planewas measured along different directions. Tab. II summarizesaccuracy assessments of the reconstructed point clouds usingdifferent similarity measures along different coordinate axesby LS plane fitting. Additionally. the mean absolute differencebetween the achieved point cloud respective to the LiDAR datais applied for total accuracy evaluation. As represented in Tab.II,the MI similarity measure gives the best results which meansthat pixel-wise MI has better performance than other classicalsimilarity measures for complicated matching scenarios suchas SAR-optical image pairs. Still, the achieved 3D accuracy isslightly worse than the pre-processed pixel size, thus indicatingthe need for more sophisticated similarity measures, which arespecifically dedicated to the SAR-optical multi-sensor case.Such a similarity measure could, e.g., result from deep learning[10].To reduce the influene of outliers, we applied some post-processing to the reconstructed point clouds, exploiting anexternal DEM such as SRTM to detect inaccurate points. Forthis, those points, whose heights deviate from SRTM by morethan a specified threshold are removed. As our assessmentsdemonstrate, using this outlier removal can significantly im-prove the quality of the obtained point cloud (cf. Tab II), inspite of the low resolution of the SRTM dataset.C. LOD1 Building Model ResultFigure 4 displays an LOD 1 city model consisting ofprismatic building models generated by combining the heightinformation derived from SAR-optical stereogrammetry andbuilding footprints provided by OpenStreetMap. The heightaccuracy of this LOD 1 model was validated by comparingit a similarly derived city model with heights taken from theLiDAR point cloud. The results illustrate that the median andmean absolute height differences between the produced SAR-optical model and the LiDAR city model are 1.56 m and0.84 m, respectively.IV. DISCUSSIONA. Potential and Limitations of SAR-Optical Dense Stere-ogrammetryThe achieved results demonstrate the possibility of 3Dreconstruction from very high-resolution SAR-optical imagepairs by stereogrammetry in urban areas. The best result wasobtained by using MI as similarity measure, while using NCCleads to a point cloud with significantly worse accuracy. Thisis partially due to the fact that area-based similarity measuressuch as NCC are not really suitable for dense image matching,since the required rectangular window also acts as some kindof smoothing operator. However, as shown in Fig. 3, the pointcloud from pixel-wise MI-based dense matching also suffersfrom the noise effects. Apart from the difficult geometricrelationship between SAR and optical imagery, their differentradiometric appearence makes the matching very challenging.Since the applied similarity measures are classic measures thatare not specifically suited for the SAR-optical multi-sensormatching case yet, this limits the matching success rate. Inaddition, it has been shown that only few scene parts arejointly visible to the SAR and the optical sensors at all intypical urban scenarios.In spite of all this, SAR-optical stereogrammetry led to apoint cloud with a median accuracy of about 1.5 to 2m andthe 25%-quantile of best points well in the sub-pixel accuracydomain, thus indicating the general feasibility of the multi-sensor stereogrammetry concept.B. Potential of Building Model Derivation from SAR-OpticalPoint CloudsThere are different challenges relevant to 3D building modelgeneration as well as an accuracy evaluation of the producedmodel which should be discussed. Firstly, the heights locatedin an outline of a building should be consistent. Some incon-sistencies come from noise produced during the SAR-opticaldense matching process. Moreover, regarding shapes of roofsparticularly inclined types, the location of points differs fromthe topmost toward the lowest part. Then, making a decision,which cluster of points are suitable for 3D reconstruction in theLOD 1 level is a challenge. In this research, we simply usedmedian as a robust estimator against the large heights changesof reconstructed points located in a footprint of building.For validating accuracy of the produced 3D model, we alsoapplied median of absolute height differences in addition tomean absolute height differences which makes the assessmentmore robust against the large heights differences. In otherwords, by using median, the heights of partially high-risebuildings are discarded due to domination of noise inducedby such as layover and shadow effects. However, employingboth metrics, the evaluation results demonstrate the potentialof SAR-optical stereogrammetry for LOD1 model generation.V. SUMMARY AND CONCLUSIONIn this research, the possibility and potential of urban3D reconstruction from TerraSAR-X and WorldView-2 imagepairs through a stereogrammetry pipeline was investigated.Conventional semi-global matching with different similaritymeasures to drive the cost function was carried out to achievepoint clouds with a median accuracy of about 1.5 to 2mand the 25%-quantile of best points well in the sub-pixelaccuracy domain. Then, SRTM data were used to removeoutliers from the point clouds and further improving the meanTABLE II: Accuracy assessment of reconstructed point clouds using different similarity measures with respect to LiDARreferenceSimilarity Measures Mean (m) STD (m) RMSE (m) Mean (m)X Y Z X Y Z X Y Z dMI 0.00 -0.04 0.27 1.57 1.69 3.09 1.57 1.69 3.10 2.75Census 0.01 -0.01 0.66 1.64 1.86 3.33 1.64 1.86 3.40 2.94NCC -0.01 -0.02 0.70 1.76 2.04 3.66 1.76 2.04 3.73 3.26MI+Census -0.01 0.00 0.61 1.62 1.78 3.25 1.62 1.78 3.31 2.86(a) LOD1 from SAR-optical stereogrammetry (b) LOD1 from LiDAR dataFig. 4: 3D prismatic model generation with an integration of height information derived from SAR-optical stereogrammetry(a) and building footprint layer from OpenStreetMap, (b): LOD1 model from LiDAR dataTABLE III: Accuracy assessment of point clouds after SRTM-based outlier removalArea Point Cloud Mean (m)Berlinoriginal 2.75filtered 2.35SRTM 2.65point cloud accuracy. Finally, simple prismatic 3D buildingmodels were generated by combining heights produced bySAR-optical stereogrammetry with footprints of buildings pro-vided by OpenStreetMap. These preliminary results demon-strate the possibility of prismatic bulding model generation(at LOD 1 level) through SAR-optical stereogrammetry andalso a possible way for removing noise from SAR-opticalstereogrammetry-derived heights.ACKNOWLEDGMENTThe authors want to thank everyone, who has providedtest data for this research: European Space Imaging for theWorldView-2 image, DLR for the TerraSAR-X images, andLand Berlin (EU EFRE project) for the LiDAR reference dataof Berlin.REFERENCES[1] M. Schmitt and X. X. Zhu, “Data fusion and remote sensing: An ever-growing relationship,” IEEE Geoscience and Remote Sensing Magazine,vol. 4, pp. 6–23, Dec 2016.[2] J. D. Wegner, J. R. Ziehn, and U. Soergel, “Combining high-resolutionoptical and InSAR features for height estimation of buildings with flatroofs,” IEEE Transactions on Geoscience and Remote Sensing, vol. 52,pp. 5840–5854, Sept 2014.[3] H. Bagheri, M. Schmitt, P. d’Angelo, and X. X. Zhu, “A frameworkfor SAR-Optical stereogrammetry over urban areas,” ISPRS Journal ofPhotogrammetry and Remote Sensing, 2018. in press.[4] H. , M. Schmitt, P. d’Angelo, and X. X. Zhu, “Exploring the applicabilityof semi-global matching for SAR-optical stereogrammetry of urbanscenes,” ISPRS - International Archives of the Photogrammetry, RemoteSensing and Spatial Information Sciences, vol. XLII-2, pp. 43–48, 2018.[5] H. Hirschmu¨ller, “Stereo processing by semiglobal matching and mutualinformation,” IEEE Transactions on Pattern Analysis and MachineIntelligence, vol. 30, pp. 328–341, Feb 2008.[6] H. Hirschmuller and D. Scharstein, “Evaluation of stereo matching costson images with radiometric differences,” IEEE Transactions on PatternAnalysis and Machine Intelligence, vol. 31, pp. 1582–1599, Sept 2009.[7] H. Arefi, From LiDAR point clouds to 3D building models. PhD thesis,Universita¨tsbibliothek der Universita¨t der Bundeswehr Mu¨nchen, 2009.[8] T. H. Kolbe, G. Gro¨ger, and L. Plu¨mer, CityGML: Interoperable Accessto 3D City Models, pp. 883–899. Berlin, Heidelberg: Springer BerlinHeidelberg, 2005.[9] Karlsruhe Institute of Technology (KIT), Institute for Applied ComputerScience, “CityGML example FZK-Haus.” http://www.citygmlwiki.org/index.php?title=FZK Haus, 2018. (Accessed 10.18).[10] L. H. Hughes, M. Schmitt, L. Mou, Y. Wang, and X. X. Zhu, “Identifyingcorresponding patches in SAR and optical images with a Pseudo-Siamese CNN,” IEEE Geoscience and Remote Sensing Letters, vol. 15,pp. 784–788, May 2018.

Towards the reconstruction of prismatic building models by SAR-optical stereogrammetry

https://elib.dlr.de/130998/1/08808985.pdf

Towards the reconstruction of prismatic building models by SAR-optical stereogrammetry

Abstract

Similar works

Full text

Available Versions

Crossref

Institute of Transport Research:Publications