Combining visibility analysis and deep learning for refinement of semantic 3D building models by conflict classification

Semantic 3D building models are widely available and used in numerous applications. Such 3D building models display rich semantics but no façade openings, chiefly owing to their aerial acquisition techniques. Hence, refining models’ façades using dense, street-level, terrestrial point clouds seems a promising strategy. In this paper, we propose a method of combining visibility analysis and neural networks for enriching 3D models with window and door features. In the method, occupancy voxels are fused with classified point clouds, which provides semantics to voxels. Voxels are also used to identify conflicts between laser observations and 3D models. The semantic voxels and conflicts are combined in a Bayesian network to classify and delineate façade openings, which are reconstructed using a 3D model library. Unaffected building semantics is preserved while the updated one is added, thereby upgrading the building model to LoD3. Moreover, Bayesian network results are back-projected onto point clouds to improve points’ classification accuracy. We tested our method on a municipal CityGML LoD2 repository and the open point cloud datasets: TUM-MLS-2016 and TUM-FAÇADE. Validation results revealed that the method improves the accuracy of point cloud semantic segmentation and upgrades buildings with façade elements. The method can be applied to enhance the accuracy of urban simulations and facilitate the development of semantic segmentation algorithms

EXTRACTION OF SOLAR CELLS FROM UAV-BASED THERMAL IMAGE SEQUENCES

This paper discusses the automatic generation of thermal infrared ortho image mosaics and the extraction of solar cells from these ortho image mosaics. Image sequences are recorded by a thermal infrared (TIR) camera mounted on a remotely piloted aerial system (RPAS). The image block is relatively oriented doing a bundle block adjustment and transferred to a local coordinate system using ground control points. The resulting ortho image mosaic is searched for solar cells. A library of templates of solar cells from thermal images is used to learn an implicit shape model. The extraction of the single solar cells is done by estimating corners and centre points of cells using these shape models in a Markov-Chain-Monte-Carlo algorithm by combining four corners and a centre point. As for the limited geometric resolution and radiometric contrast, most of the cells are not directly detected. An iterative process based on the knowledge of the regular grid structure of a solar cell installation is used to predict further cells and verify their existence by repeating the corner extraction and grammar combination. Results show that this work flow is able to detect most of the solar cells under the condition that the cells have a more or less common radiometric behaviour and no reflections i.e. from the sun occur. The cells need a rectangular shape and have the same orientation so that the model of the grammar is applicable to the solar cells

A Landmark Selection Method for Object-Based Visual Outdoor Localization Approaches of Automated Ground Vehicles

Autonomous vehicles must navigate independently in an outdoor environment using features or objects. However, some objects may be more or less suitable for localization due to their attributes. Therefore, this work investigates the suitability of landmarks for camera- and object-based outdoor localization methods. First, object attributes are methodically derived from the requirements of object-based localization. The physical representation on the camera image plane, probability of occurrence, and persistence were identified as influencing the object localization suitability. The influence of the object’s camera image plane representation regarding object recognition algorithms is not considered or discussed, but advice on the minimum object pixel size is provided. The first milestone was the creation of an equation for object localization suitability calculation by normalizing and multiplying the identified attributes. Simultaneously, potential objects from the outdoor environment were identified, resulting in a structured object catalog. The results of the equation and catalog are a ranked according to the object localization suitability in a comparison table. Our comparison demonstrates that objects such as buildings or trees are more suitable than street lane markings for self-localization. However, most current datasets do not include the proposed instantiated objects. The paper addresses this issue, assists in the object selection for outdoor localization methods and provides input for the creation of future-oriented datasets and autonomous driving maps

REGISTRATION OF UAV DATA AND ALS DATA USING POINT TO DEM DISTANCES FOR BATHYMETRIC CHANGE DETECTION

This paper shows a method to register point clouds from images of UAV-mounted airborne cameras as well as airborne laser scanner data. The focus is a general technique which does rely neither on linear or planar structures nor on the point cloud density. Therefore, the proposed approach is also suitable for rural areas and water bodies captured via different sensor configurations. This approach is based on a regular 2.5D grid generated from the segmented ground points of the 3D point cloud. It is assumed that initial values for the registration are already estimated, e.g. by measured exterior orientation parameters with the UAV mounted GNSS and IMU. These initial parameters are finely tuned by minimizing the distances between the 3D points of a target point cloud to the generated grid of the source point cloud in an iteration process. To eliminate outliers (e.g., vegetation points) a threshold for the distances is defined dynamically at each iteration step, which filters ground points during the registration. The achieved accuracy of the registration is up to 0.4 m in translation and up to 0.3 degrees in rotation, by using a raster size of the DEM of 2 m. Considering the ground sampling distance of the airborne data which is up to 0.4 m between the scan lines, this result is comparable to the result achieved by an ICP algorithm, but the proposed approach does not rely on point densities and is therefore able to solve registrations where the ICP have difficulties

FUSION OF 3D POINT CLOUDS WITH TIR IMAGES FOR INDOOR SCENE RECONSTRUCTION

Obtaining accurate 3D descriptions in the thermal infrared (TIR) is a quite challenging task due to the low geometric resolutions of TIR cameras and the low number of strong features in TIR images. Combining the radiometric information of the thermal infrared with 3D data from another sensor is able to overcome most of the limitations in the 3D geometric accuracy. In case of dynamic scenes with moving objects or a moving sensor system, a combination with RGB cameras and profile laserscanners is suitable. As a laserscanner is an active sensor in the visible red or near infrared (NIR) and the thermal infrared camera captures the radiation emitted by the objects in the observed scene, the combination of these two sensors for close range applications are independent from external illumination or textures in the scene. This contribution focusses on the fusion of point clouds from terrestrial laserscanners and RGB cameras with images from thermal infrared mounted together on a robot for indoor 3D reconstruction. The system is geometrical calibrated including the lever arm between the different sensors. As the field of view is different for the sensors, the different sensors record the same scene points not exactly at the same time. Thus, the 3D scene points of the laserscanner and the photogrammetric point cloud from the RGB camera have to be synchronized before point cloud fusion and adding the thermal channel to the 3D points

Analyzing the impact of semantic LoD3 building models on image-based vehicle localization

Numerous navigation applications rely on data from global navigation satellite systems (GNSS), even though their accuracy is compromised in urban areas, posing a significant challenge, particularly for precise autonomous car localization. Extensive research has focused on enhancing localization accuracy by integrating various sensor types to address this issue. This paper introduces a novel approach for car localization, leveraging image features that correspond with highly detailed semantic 3D building models. The core concept involves augmenting positioning accuracy by incorporating prior geometric and semantic knowledge into calculations. The work assesses outcomes using Level of Detail 2 (LoD2) and Level of Detail 3 (LoD3) models, analyzing whether facade-enriched models yield superior accuracy. This comprehensive analysis encompasses diverse methods, including off-the-shelf feature matching and deep learning, facilitating thorough discussion. Our experiments corroborate that LoD3 enables detecting up to 69% more features than using LoD2 models. We believe that this study will contribute to the research of enhancing positioning accuracy in GNSS-denied urban canyons. It also shows a practical application of under-explored LoD3 building models on map-based car positioning