Continuous Modeling of 3D Building Rooftops From Airborne LIDAR and Imagery

In recent years, a number of mega-cities have provided 3D photorealistic virtual models to support the decisions making process for maintaining the cities' infrastructure and environment more effectively. 3D virtual city models are static snap-shots of the environment and represent the status quo at the time of their data acquisition. However, cities are dynamic system that continuously change over time. Accordingly, their virtual representation need to be regularly updated in a timely manner to allow for accurate analysis and simulated results that decisions are based upon. The concept of "continuous city modeling" is to progressively reconstruct city models by accommodating their changes recognized in spatio-temporal domain, while preserving unchanged structures. However, developing a universal intelligent machine enabling continuous modeling still remains a challenging task. Therefore, this thesis proposes a novel research framework for continuously reconstructing 3D building rooftops using multi-sensor data. For achieving this goal, we first proposes a 3D building rooftop modeling method using airborne LiDAR data. The main focus is on the implementation of an implicit regularization method which impose a data-driven building regularity to noisy boundaries of roof planes for reconstructing 3D building rooftop models. The implicit regularization process is implemented in the framework of Minimum Description Length (MDL) combined with Hypothesize and Test (HAT). Secondly, we propose a context-based geometric hashing method to align newly acquired image data with existing building models. The novelty is the use of context features to achieve robust and accurate matching results. Thirdly, the existing building models are refined by newly proposed sequential fusion method. The main advantage of the proposed method is its ability to progressively refine modeling errors frequently observed in LiDAR-driven building models. The refinement process is conducted in the framework of MDL combined with HAT. Markov Chain Monte Carlo (MDMC) coupled with Simulated Annealing (SA) is employed to perform a global optimization. The results demonstrates that the proposed continuous rooftop modeling methods show a promising aspects to support various critical decisions by not only reconstructing 3D rooftop models accurately, but also by updating the models using multi-sensor data

3D Modelling from Real Data

The genesis of a 3D model has basically two definitely different paths. Firstly we can consider the CAD generated models, where the shape is defined according to a user drawing action, operating with different mathematical “bricks” like B-Splines, NURBS or subdivision surfaces (mathematical CAD modelling), or directly drawing small polygonal planar facets in space, approximating with them complex free form shapes (polygonal CAD modelling). This approach can be used for both ideal elements (a project, a fantasy shape in the mind of a designer, a 3D cartoon, etc.) or for real objects. In the latter case the object has to be first surveyed in order to generate a drawing coherent with the real stuff. If the surveying process is not only a rough acquisition of simple distances with a substantial amount of manual drawing, a scene can be modelled in 3D by capturing with a digital instrument many points of its geometrical features and connecting them by polygons to produce a 3D result similar to a polygonal CAD model, with the difference that the shape generated is in this case an accurate 3D acquisition of a real object (reality-based polygonal modelling). Considering only device operating on the ground, 3D capturing techniques for the generation of reality-based 3D models may span from passive sensors and image data (Remondino and El-Hakim, 2006), optical active sensors and range data (Blais, 2004; Shan & Toth, 2008; Vosselman and Maas, 2010), classical surveying (e.g. total stations or Global Navigation Satellite System - GNSS), 2D maps (Yin et al., 2009) or an integration of the aforementioned methods (Stumpfel et al., 2003; Guidi et al., 2003; Beraldin, 2004; Stamos et al., 2008; Guidi et al., 2009a; Remondino et al., 2009; Callieri et al., 2011). The choice depends on the required resolution and accuracy, object dimensions, location constraints, instrument’s portability and usability, surface characteristics, working team experience, project’s budget, final goal, etc. Although aware of the potentialities of the image-based approach and its recent developments in automated and dense image matching for non-expert the easy usability and reliability of optical active sensors in acquiring 3D data is generally a good motivation to decline image-based approaches. Moreover the great advantage of active sensors is the fact that they deliver immediately dense and detailed 3D point clouds, whose coordinate are metrically defined. On the other hand image data require some processing and a mathematical formulation to transform the two-dimensional image measurements into metric three-dimensional coordinates. Image-based modelling techniques (mainly photogrammetry and computer vision) are generally preferred in cases of monuments or architectures with regular geometric shapes, low budget projects, good experience of the working team, time or location constraints for the data acquisition and processing. This chapter is intended as an updated review of reality-based 3D modelling in terrestrial applications, with the different categories of 3D sensing devices and the related data processing pipelines

Relating Multimodal Imagery Data in 3D

This research develops and improves the fundamental mathematical approaches and techniques required to relate imagery and imagery derived multimodal products in 3D. Image registration, in a 2D sense, will always be limited by the 3D effects of viewing geometry on the target. Therefore, effects such as occlusion, parallax, shadowing, and terrain/building elevation can often be mitigated with even a modest amounts of 3D target modeling. Additionally, the imaged scene may appear radically different based on the sensed modality of interest; this is evident from the differences in visible, infrared, polarimetric, and radar imagery of the same site. This thesis develops a `model-centric\u27 approach to relating multimodal imagery in a 3D environment. By correctly modeling a site of interest, both geometrically and physically, it is possible to remove/mitigate some of the most difficult challenges associated with multimodal image registration. In order to accomplish this feat, the mathematical framework necessary to relate imagery to geometric models is thoroughly examined. Since geometric models may need to be generated to apply this `model-centric\u27 approach, this research develops methods to derive 3D models from imagery and LIDAR data. Of critical note, is the implementation of complimentary techniques for relating multimodal imagery that utilize the geometric model in concert with physics based modeling to simulate scene appearance under diverse imaging scenarios. Finally, the often neglected final phase of mapping localized image registration results back to the world coordinate system model for final data archival are addressed. In short, once a target site is properly modeled, both geometrically and physically, it is possible to orient the 3D model to the same viewing perspective as a captured image to enable proper registration. If done accurately, the synthetic model\u27s physical appearance can simulate the imaged modality of interest while simultaneously removing the 3-D ambiguity between the model and the captured image. Once registered, the captured image can then be archived as a texture map on the geometric site model. In this way, the 3D information that was lost when the image was acquired can be regained and properly related with other datasets for data fusion and analysis

Map-Based Localization for Unmanned Aerial Vehicle Navigation

Unmanned Aerial Vehicles (UAVs) require precise pose estimation when navigating in indoor and GNSS-denied / GNSS-degraded outdoor environments. The possibility of crashing in these environments is high, as spaces are confined, with many moving obstacles. There are many solutions for localization in GNSS-denied environments, and many different technologies are used. Common solutions involve setting up or using existing infrastructure, such as beacons, Wi-Fi, or surveyed targets. These solutions were avoided because the cost should be proportional to the number of users, not the coverage area. Heavy and expensive sensors, for example a high-end IMU, were also avoided. Given these requirements, a camera-based localization solution was selected for the sensor pose estimation. Several camera-based localization approaches were investigated. Map-based localization methods were shown to be the most efficient because they close loops using a pre-existing map, thus the amount of data and the amount of time spent collecting data are reduced as there is no need to re-observe the same areas multiple times. This dissertation proposes a solution to address the task of fully localizing a monocular camera onboard a UAV with respect to a known environment (i.e., it is assumed that a 3D model of the environment is available) for the purpose of navigation for UAVs in structured environments. Incremental map-based localization involves tracking a map through an image sequence. When the map is a 3D model, this task is referred to as model-based tracking. A by-product of the tracker is the relative 3D pose (position and orientation) between the camera and the object being tracked. State-of-the-art solutions advocate that tracking geometry is more robust than tracking image texture because edges are more invariant to changes in object appearance and lighting. However, model-based trackers have been limited to tracking small simple objects in small environments. An assessment was performed in tracking larger, more complex building models, in larger environments. A state-of-the art model-based tracker called ViSP (Visual Servoing Platform) was applied in tracking outdoor and indoor buildings using a UAVs low-cost camera. The assessment revealed weaknesses at large scales. Specifically, ViSP failed when tracking was lost, and needed to be manually re-initialized. Failure occurred when there was a lack of model features in the cameras field of view, and because of rapid camera motion. Experiments revealed that ViSP achieved positional accuracies similar to single point positioning solutions obtained from single-frequency (L1) GPS observations standard deviations around 10 metres. These errors were considered to be large, considering the geometric accuracy of the 3D model used in the experiments was 10 to 40 cm. The first contribution of this dissertation proposes to increase the performance of the localization system by combining ViSP with map-building incremental localization, also referred to as simultaneous localization and mapping (SLAM). Experimental results in both indoor and outdoor environments show sub-metre positional accuracies were achieved, while reducing the number of tracking losses throughout the image sequence. It is shown that by integrating model-based tracking with SLAM, not only does SLAM improve model tracking performance, but the model-based tracker alleviates the computational expense of SLAMs loop closing procedure to improve runtime performance. Experiments also revealed that ViSP was unable to handle occlusions when a complete 3D building model was used, resulting in large errors in its pose estimates. The second contribution of this dissertation is a novel map-based incremental localization algorithm that improves tracking performance, and increases pose estimation accuracies from ViSP. The novelty of this algorithm is the implementation of an efficient matching process that identifies corresponding linear features from the UAVs RGB image data and a large, complex, and untextured 3D model. The proposed model-based tracker improved positional accuracies from 10 m (obtained with ViSP) to 46 cm in outdoor environments, and improved from an unattainable result using VISP to 2 cm positional accuracies in large indoor environments. The main disadvantage of any incremental algorithm is that it requires the camera pose of the first frame. Initialization is often a manual process. The third contribution of this dissertation is a map-based absolute localization algorithm that automatically estimates the camera pose when no prior pose information is available. The method benefits from vertical line matching to accomplish a registration procedure of the reference model views with a set of initial input images via geometric hashing. Results demonstrate that sub-metre positional accuracies were achieved and a proposed enhancement of conventional geometric hashing produced more correct matches - 75% of the correct matches were identified, compared to 11%. Further the number of incorrect matches was reduced by 80%

Integration of LiDAR and photogrammetric data for enhanced aerial triangulation and camera calibration

PhD ThesisThe integration of complementary airborne light detection and ranging (LiDAR) and photogrammetric data continues to receive attention from the relevant research communities. Such an approach requires the optimized registration of the two data types within a common coordinate reference frame and thus enables the cross-calibration of one information source against another. This research assumes airborne LiDAR as a reference dataset against which in-flight camera system calibration and validation can be performed. The novel methodology involves the production of dense photogrammetric point clouds derived using the simultaneous adjustment of GNSS/IMU data and a dense set of photogrammetric tie points. Quality of the generated photogrammetric dataset is further improved through introducing the self-calibration additional parameters in the combined adjustment. A robust least squares surface matching algorithm is then used to minimise the Euclidean distances between the two datasets. After successful matching, well distributed LiDAR-derived control points (LCPs) are automatically identified and extracted. Adjustment of the photogrammetric data is then repeated using extracted LCPs in a self-calibrating bundle adjustment. The research methodology was tested using two datasets acquired using different photogrammetric digital sensor systems, a Microsoft UltraCamX large format camera and an Applanix DSS322 medium format camera. Systematic sensitivity testing included the influence of the number and weighting of LCPs required to achieve optimised adjustment. For the UltraCamX block it was found that when the number of control points exceeded 80, the accuracy of the adjustment stabilized at c. 2 cm in all axes, regardless of point weighting. Results were also compared with those from reference calibration using surveyed ground control points in the test area, with good agreement found between the two. Similar results were obtained for the DSS322 block, with block accuracy stabilizing at 100 LCPs. Moreover, for the DSS322 camera, introducing self-calibration greatly improved the accuracy of aerial triangulation

Orientation and integration of images and image blocks with laser scanning data

Laser scanning and photogrammetry are methods for effective and accurate measurement and classification of urban and forest areas. Because these methods complement each other, then integration or integrated use brings additional benefits to real-life applications. However, finding tie features between data sets is a challenging task since laser scanning and imagery are far from each other in nature. The aim of this thesis was to create methods for solving relative orientations between laser scanning data and imagery that would assist in near-future applications integrating laser scanning and photogrammetry. Moreover, a further goal was to create methods enabling the use of data acquired from very different perspectives, such as terrestrial and airborne data. To meet these aims, an interactive orientation method enabling the use of single images, stereo images or larger image blocks was developed and tested. The multi-view approach usually has a significant advantage over the use of a single image. After accurate orientation of laser scanning data and imagery, versatile applications become available. Such applications include, e.g., automatic object recognition, accurate classification of individual trees, point cloud densification, automatic classification of land use, system calibration, and generation of photorealistic 3D models. Besides the orientation part, another aim of the research was to investigate how to fuse or use these two data types together in applications. As a result, examples that evaluated the behavior of laser point clouds in both urban and forestry areas, detection and visualization of temporal changes, enhanced data understanding, stereo visualization, multi-source and multi-angle data fusion, point cloud colorizing, and detailed examination of full waveform laser scanning data were given

Automated 3D object modeling from aerial video imagery

Research in physically accurate 3D modeling of a scene is gaining momentum because of its far reaching applications in civilian and defense sectors. The modeled 3D scene must conform both geometrically and spectrally to the real world for all the applications. Geometric modeling of a scene can be achieved in many ways of which the two most popular methods are - a) using multiple 2D passive images of the scene also called as stereo vision and b) using 3D point clouds like Lidar (Light detection and ranging) data. In this research work, we derive the 3D models of objects in a scene using passive aerial video imagery. At present, this geometric modeling requires a lot of manual intervention due to a variety of factors like sensor noise, low contrast conditions during image capture, etc. Hence long time periods, in the order of weeks and months, are required to model even a small scene. This thesis focuses on automating the process of geometric modeling of objects in a scene from passive aerial video imagery. The aerial video frames are stitched into stereo mosaics. These stereo mosaics not only provide the elevation information of a scene but also act as good 3D visualization tools. The 3D information obtained from the stereo mosaics is used to identify the various 3D objects, especially man-made buildings using probabilistic inference provided by Bayesian Networks. The initial 3D building models are further optimized by projecting them on to the individual video frames. The limitations of the state-of-art technology in attaining these goals are presented along with the techniques to overcome them. The improvement that can be achieved in the accuracy of the 3D models when Lidar data is fused with aerial video during the object identification process is also examined

UAVs for the Environmental Sciences

This book gives an overview of the usage of UAVs in environmental sciences covering technical basics, data acquisition with different sensors, data processing schemes and illustrating various examples of application