External multi-modal imaging sensor calibration for sensor fusion: A review

Multi-modal data fusion has gained popularity due to its diverse applications, leading to an increased demand for external sensor calibration. Despite several proven calibration solutions, they fail to fully satisfy all the evaluation criteria, including accuracy, automation, and robustness. Thus, this review aims to contribute to this growing field by examining recent research on multi-modal imaging sensor calibration and proposing future research directions. The literature review comprehensively explains the various characteristics and conditions of different multi-modal external calibration methods, including traditional motion-based calibration and feature-based calibration. Target-based calibration and targetless calibration are two types of feature-based calibration, which are discussed in detail. Furthermore, the paper highlights systematic calibration as an emerging research direction. Finally, this review concludes crucial factors for evaluating calibration methods and provides a comprehensive discussion on their applications, with the aim of providing valuable insights to guide future research directions. Future research should focus primarily on the capability of online targetless calibration and systematic multi-modal sensor calibration.Ministerio de Ciencia, Innovación y Universidades | Ref. PID2019-108816RB-I0

COMPARISON OF LOW COST PHOTOGRAMMETRIC SURVEY WITH TLS AND LEICA PEGASUS BACKPACK 3D MODELS

This paper considers Leica backpack and photogrammetric surveys of a mediaeval bastion in Padua, Italy. Furhtermore, terrestrial laser scanning (TLS) survey is considered in order to provide a state of the art reconstruction of the bastion. Despite control points are typically used to avoid deformations in photogrammetric surveys and ensure correct scaling of the reconstruction, in this paper a different approach is considered: this work is part of a project aiming at the development of a system exploiting ultra-wide band (UWB) devices to provide correct scaling of the reconstruction. In particular, low cost Pozyx UWB devices are used to estimate camera positions during image acquisitions. Then, in order to obtain a metric reconstruction, scale factor in the photogrammetric survey is estimated by comparing camera positions obtained from UWB measurements with those obtained from photogrammetric reconstruction. Compared with the TLS survey, the considered photogrammetric model of the bastion results in a RMSE of 21.9cm, average error 13.4cm, and standard deviation 13.5cm. Excluding the final part of the bastion left wing, where the presence of several poles make reconstruction more difficult, (RMSE) fitting error is 17.3cm, average error 11.5cm, and standard deviation 9.5cm. Instead, comparison of Leica backpack and TLS surveys leads to an average error of 4.7cm and standard deviation 0.6cm (4.2 cm and 0.3 cm, respectively, by excluding the final part of the left wing)

Integration of LiDAR and photogrammetric data for enhanced aerial triangulation and camera calibration

PhD ThesisThe integration of complementary airborne light detection and ranging (LiDAR) and photogrammetric data continues to receive attention from the relevant research communities. Such an approach requires the optimized registration of the two data types within a common coordinate reference frame and thus enables the cross-calibration of one information source against another. This research assumes airborne LiDAR as a reference dataset against which in-flight camera system calibration and validation can be performed. The novel methodology involves the production of dense photogrammetric point clouds derived using the simultaneous adjustment of GNSS/IMU data and a dense set of photogrammetric tie points. Quality of the generated photogrammetric dataset is further improved through introducing the self-calibration additional parameters in the combined adjustment. A robust least squares surface matching algorithm is then used to minimise the Euclidean distances between the two datasets. After successful matching, well distributed LiDAR-derived control points (LCPs) are automatically identified and extracted. Adjustment of the photogrammetric data is then repeated using extracted LCPs in a self-calibrating bundle adjustment. The research methodology was tested using two datasets acquired using different photogrammetric digital sensor systems, a Microsoft UltraCamX large format camera and an Applanix DSS322 medium format camera. Systematic sensitivity testing included the influence of the number and weighting of LCPs required to achieve optimised adjustment. For the UltraCamX block it was found that when the number of control points exceeded 80, the accuracy of the adjustment stabilized at c. 2 cm in all axes, regardless of point weighting. Results were also compared with those from reference calibration using surveyed ground control points in the test area, with good agreement found between the two. Similar results were obtained for the DSS322 block, with block accuracy stabilizing at 100 LCPs. Moreover, for the DSS322 camera, introducing self-calibration greatly improved the accuracy of aerial triangulation

Machine-human Cooperative Control of Welding Process

An innovative auxiliary control system is developed to cooperate with an unskilled welder in a manual GTAW in order to obtain a consistent welding performance. In the proposed system, a novel mobile sensing system is developed to non-intrusively monitor a manual GTAW by measuring three-dimensional (3D) weld pool surface. Specifically, a miniature structured-light laser amounted on torch projects a dot matrix pattern on weld pool surface during the process; Reflected by the weld pool surface, the laser pattern is intercepted by and imaged on the helmet glass, and recorded by a compact camera on it. Deformed reflection pattern contains the geometry information of weld pool, thus is utilized to reconstruct its $3$D surface. An innovative image processing algorithm and a reconstruction scheme have been developed for (3D) reconstruction. The real-time spatial relations of the torch and the helmet is formulated during welding. Two miniature wireless inertial measurement units (WIMU) are mounted on the torch and the helmet, respectively, to detect their rotation rates and accelerations. A quaternion based unscented Kalman filter (UKF) has been designed to estimate the helmet/torch orientations based on the data from the WIMUs. The distance between the torch and the helmet is measured using an extra structure-light low power laser pattern. Furthermore, human welder\u27s behavior in welding performance has been studied, e.g., a welder`s adjustments on welding current were modeled as response to characteristic parameters of the three-dimensional weld pool surface. This response model as a controller is implemented both automatic and manual gas tungsten arc welding process to maintain a consistent full penetration

Accurate Calibration Scheme for a Multi-Camera Mobile Mapping System

Mobile mapping systems (MMS) are increasingly used for many photogrammetric and computer vision applications, especially encouraged by the fast and accurate geospatial data generation. The accuracy of point position in an MMS is mainly dependent on the quality of calibration, accuracy of sensor synchronization, accuracy of georeferencing and stability of geometric configuration of space intersections. In this study, we focus on multi-camera calibration (interior and relative orientation parameter estimation) and MMS calibration (mounting parameter estimation). The objective of this study was to develop a practical scheme for rigorous and accurate system calibration of a photogrammetric mapping station equipped with a multi-projective camera (MPC) and a global navigation satellite system (GNSS) and inertial measurement unit (IMU) for direct georeferencing. The proposed technique is comprised of two steps. Firstly, interior orientation parameters of each individual camera in an MPC and the relative orientation parameters of each cameras of the MPC with respect to the first camera are estimated. In the second step the offset and misalignment between MPC and GNSS/IMU are estimated. The global accuracy of the proposed method was assessed using independent check points. A correspondence map for a panorama is introduced that provides metric information. Our results highlight that the proposed calibration scheme reaches centimeter-level global accuracy for 3D point positioning. This level of global accuracy demonstrates the feasibility of the proposed technique and has the potential to fit accurate mapping purposes

The Visual–Inertial Canoe Dataset

We present a dataset collected from a canoe along the Sangamon River in Illinois. The canoe was equipped with a stereo camera, an inertial measurement unit (IMU), and a global positioning system (GPS) device, which provide visual data suitable for stereo or monocular applications, inertial measurements, and position data for ground truth. We recorded a canoe trip up and down the river for 44 minutes covering a 2.7 km round trip. The dataset adds to those previously recorded in unstructured environments and is unique in that it is recorded on a river, which provides its own set of challenges and constraints that are described in this paper. The dataset is stored on the Illinois Data Bank and can be accessed at: https://doi.org/10.13012/B2IDB-9342111_V1

A novel visual pipework inspection system

The interior visual inspection of pipelines in the nuclear industry is a safety critical activity conducted during outages to ensure the continued safe and reliable operation of plant. Typically, the video output by a manually deployed probe is viewed by an operator looking to identify and localise surface defects such as corrosion, erosion and pitting. However, it is very challenging to estimate the nature and extent of defects by viewing a large structure through a relatively small field of view. This work describes a new visual inspection system employing photogrammetry using a fisheye camera and a structured light system to map the internal geometry of pipelines by generating a photorealistic, geometrically accurate surface model. The error of the system output was evaluated through comparison to a ground truth laser scan (ATOS GOM Triple Scan) of a nuclear grade split pipe sample (stainless steel 304L, 80mm internal diameter) containing defects representative of the application – the error was found to be submillimetre across the sample