Semi-automatic 3D reconstruction of urban areas using epipolar geometry and template matching

WOS:000240143800002 (Nº de Acesso Web of Science)In this work we describe a novel technique for semi-automatic three-dimensional (3D) reconstruction of urban areas, from airborne stereo-pair images whose output is VRML or DXF. The main challenge is to compute the relevant information—building's height and volume, roof's description, and texture—algorithmically, because it is very time consuming and thus expensive to produce it manually for large urban areas. The algorithm requires some initial calibration input and is able to compute the above-mentioned building characteristics from the stereo pair and the availability of the 2D CAD and the digital elevation model of the same area, with no knowledge of the camera pose or its intrinsic parameters. To achieve this, we have used epipolar geometry, homography computation, automatic feature extraction and we have solved the feature correspondence problem in the stereo pair, by using template matching

Towards Automatic SAR-Optical Stereogrammetry over Urban Areas using Very High Resolution Imagery

In this paper we discuss the potential and challenges regarding SAR-optical stereogrammetry for urban areas, using very-high-resolution (VHR) remote sensing imagery. Since we do this mainly from a geometrical point of view, we first analyze the height reconstruction accuracy to be expected for different stereogrammetric configurations. Then, we propose a strategy for simultaneous tie point matching and 3D reconstruction, which exploits an epipolar-like search window constraint. To drive the matching and ensure some robustness, we combine different established handcrafted similarity measures. For the experiments, we use real test data acquired by the Worldview-2, TerraSAR-X and MEMPHIS sensors. Our results show that SAR-optical stereogrammetry using VHR imagery is generally feasible with 3D positioning accuracies in the meter-domain, although the matching of these strongly hetereogeneous multi-sensor data remains very challenging. Keywords: Synthetic Aperture Radar (SAR), optical images, remote sensing, data fusion, stereogrammetr

Refined Equivalent Pinhole Model for Large-scale 3D Reconstruction from Spaceborne CCD Imagery

In this study, we present a large-scale earth surface reconstruction pipeline for linear-array charge-coupled device (CCD) satellite imagery. While mainstream satellite image-based reconstruction approaches perform exceptionally well, the rational functional model (RFM) is subject to several limitations. For example, the RFM has no rigorous physical interpretation and differs significantly from the pinhole imaging model; hence, it cannot be directly applied to learning-based 3D reconstruction networks and to more novel reconstruction pipelines in computer vision. Hence, in this study, we introduce a method in which the RFM is equivalent to the pinhole camera model (PCM), meaning that the internal and external parameters of the pinhole camera are used instead of the rational polynomial coefficient parameters. We then derive an error formula for this equivalent pinhole model for the first time, demonstrating the influence of the image size on the accuracy of the reconstruction. In addition, we propose a polynomial image refinement model that minimizes equivalent errors via the least squares method. The experiments were conducted using four image datasets: WHU-TLC, DFC2019, ISPRS-ZY3, and GF7. The results demonstrated that the reconstruction accuracy was proportional to the image size. Our polynomial image refinement model significantly enhanced the accuracy and completeness of the reconstruction, and achieved more significant improvements for larger-scale images.Comment: 24 page

3D detection of roof sections from a single satellite image and application to LOD2-building reconstruction

Reconstructing urban areas in 3D out of satellite raster images has been a long-standing and challenging goal of both academical and industrial research. The rare methods today achieving this objective at a Level Of Details $2$ rely on procedural approaches based on geometry, and need stereo images and/or LIDAR data as input. We here propose a method for urban 3D reconstruction named KIBS(\textit{Keypoints Inference By Segmentation}), which comprises two novel features: i) a full deep learning approach for the 3D detection of the roof sections, and ii) only one single (non-orthogonal) satellite raster image as model input. This is achieved in two steps: i) by a Mask R-CNN model performing a 2D segmentation of the buildings' roof sections, and after blending these latter segmented pixels within the RGB satellite raster image, ii) by another identical Mask R-CNN model inferring the heights-to-ground of the roof sections' corners via panoptic segmentation, unto full 3D reconstruction of the buildings and city. We demonstrate the potential of the KIBS method by reconstructing different urban areas in a few minutes, with a Jaccard index for the 2D segmentation of individual roof sections of $88.55\%$ and $75.21\%$ on our two data sets resp., and a height's mean error of such correctly segmented pixels for the 3D reconstruction of $1.60$ m and $2.06$ m on our two data sets resp., hence within the LOD2 precision range

Utilizzo di Immagini Acquisite da Drone Aereo per la Ricostruzione Tridimensionale Realistica di un'Area di Interesse in Tempo Reale

Tridimensional reconstruction of areas and objects of interest from images acquired from aerial drones plays a key role in numerous fields, from the less critical, like urban planning and archeological area survey, to the most critical, like natural disaster monitoring, and search & rescue. Many techniques proposed to deal with this issue require expensive instrumentation or use computationally costly algorithm or need some a priori information that are not always available. Thus, they are not suitable for certain applications or in some specific situations. In order to overcome these limitations, we designed a technique that allows automatically creating 3D models in real-time using only a couple of images of the scene one is interested in, acquired with a cheap compact camera mounted on the drone, without the needing for any additional information. The proposed technique outputs dense true-color 3D models, which give the impression to the user to be physically present within the scene. Tested for monitoring the progress of the works in a construction site, the technique has been capable to create realistic and easy-to-interpret 3D models of areas and objects in less than 1 second and with a sufficient accuracy to permit large-scale surveys

Path-Planning for optimal coverage under security constraints

Treball fet a la Technische Universität Berlin. Fakultät Elektrotechnik und Informatik[ANGLÈS] In recent years, three-dimensional building reconstruction has been an active area of research, partly motivated by the spread of low cost unmanned aerial vehicles platforms. These permit exploiting the entire three-dimensional space as long as it is free of obstacles. Current approaches manually plan a set of viewpoints from which to conduct multiple scans of a target building, and then later select the best ones to use in a structure from motion system. This procedure often has two problems: some parts are covered with low detail or some parts are evenly uncovered. In these situations, an automatic view planner is necessary; it will completely cover a building surface, while reducing time and cost of the overall process. This thesis presents an automatic view planner for three-dimensional building reconstruction based on dividing edifices into several slices and for each one solve a two-dimensional problem. From a rough model of the environment and a desired detail level, both described in a cost function, the system computes a route in which there is a set of viewpoints to completely cover a target building surface of any shape, taking into account that there may be obstacles in the environment. The final route is proposed to be followed by an unmanned aerial vehicle equipped with a digital camera.[CASTELLÀ] En los últimos años, la reconstrucción 3D de edificios ha sido un área activa de investigación, en parte motivada por la difusión de plataformas económicas de unmanned aerial vehicles. Estos permiten explotar completamente el espacio 3D mientras esté libre de obstáculos. Las soluciones actuales planean manualmente una serie de puntos desde donde realizar escaneos de un edificio objetivo, para luego seleccionar los mejores para utilizar en un sistema structure from motion. Este procedimiento a menudo tiene dos grandes problemas: algunas partes del edificio se cubren con bajo detalle u otras partes incluso no se cubren. En estas situaciones un view planner automático es necesario. Este cubrirá completamente la superficie del edificio, a la vez que reducirá costes y tiempo en el proceso global. Este proyecto presenta un view planner automático para la reconstrucción 3D de edificios basado en dividir estos en rebanadas y para cada una resolver un problema en 2D. A partir de un modelo en bruto de la escena, y un detalle deseado, ambos descritos en una función de coste, el sistema calcula una ruta en la que hay una serie de puntos que cubren completamente la superficie de un edificio objetivo de cualquier forma, teniendo en cuenta que puede haber obstáculos en la escena. La idea es que un unmanned aerial vehicle equipado con una cámara digital siga el camino final diseñado.[CATALÀ] Els darrers anys, la reconstrucció 3D d’edificis ha estat una área activa de recerca, en part motivada per la difusió de plataformes econòmiques de unmanned aerial vehicles. Aquests permeten explotar completament l’espai 3D mentres estigui lliure d’obstacles. Les solucions actuals planegen manualment una sèrie de punts des don realitzar escanejos d’un edifici objectiu, per després seleccionar els millors per a utilitzar en un sistema structure from motion. Aquest procediment sovint té dos grans problemes: algunes parts de l’edifici es cobreixen amb baix detall o altres parts inclús no es cobreixen. En aquestes situacions un view planner automàtic es necessari. Aquest cobrirà completament la superfície de l’edifici, a la vegada que reduirà costos i temps en el procés global. Aquest projecte presenta un view planner automàtic per a la reconstrucció 3D d’edificis basat en dividir aquests en llesques i per a cada una resoldre un problema en 2D. A partir d’un model en brut de l’escena, i un detall desitjat, ambdós descrits en una funció de cost, el sistema calcula una ruta en la qual hi ha una serie de punts que cobriran completament la superfície d’un edifici objectiu de qualsevol forma, tenint en compte que hi poden haver obstacles a l’escena. La idea es que un unmanned aerial vehicle equipat amb una càmera digital segueixi el camí final dissenyat

A New Approach for Realistic 3D Reconstruction of Planar Surfaces from Laser Scanning Data and Imagery Collected Onboard Modern Low-Cost Aerial Mapping Systems

Over the past few years, accurate 3D surface reconstruction using remotely-sensed data has been recognized as a prerequisite for different mapping, modelling, and monitoring applications. To fulfill the needs of these applications, necessary data are generally collected using various digital imaging systems. Among them, laser scanners have been acknowledged as a fast, accurate, and flexible technology for the acquisition of high density 3D spatial data. Despite their quick accessibility, the acquired 3D data using these systems does not provide semantic information about the nature of scanned surfaces. Hence, reliable processing techniques are employed to extract the required information for 3D surface reconstruction. Moreover, the extracted information from laser scanning data cannot be effectively utilized due to the lack of descriptive details. In order to provide a more realistic and accurate perception of the scanned scenes using laser scanning systems, a new approach for 3D reconstruction of planar surfaces is introduced in this paper. This approach aims to improve the interpretability of the extracted planar surfaces from laser scanning data using spectral information from overlapping imagery collected onboard modern low-cost aerial mapping systems, which are widely adopted nowadays. In this approach, the scanned planar surfaces using laser scanning systems are initially extracted through a novel segmentation procedure, and then textured using the acquired overlapping imagery. The implemented texturing technique, which intends to overcome the computational inefficiency of the previously-developed 3D reconstruction techniques, is performed in three steps. In the first step, the visibility of the extracted planar surfaces from laser scanning data within the collected images is investigated and a list of appropriate images for texturing each surface is established. Successively, an occlusion detection procedure is carried out to identify the occluded parts of these surfaces in the field of view of captured images. In the second step, visible/non-occluded parts of the planar surfaces are decomposed into segments that will be textured using individual images. Finally, a rendering procedure is accomplished to texture these parts using available images. Experimental results from overlapping laser scanning data and imagery collected onboard aerial mapping systems verify the feasibility of the proposed approach for efficient realistic 3D surface reconstruction

車載画像を用いた都市の4次元モデリング

Tohoku University岡谷貴之課