Deep Thermal Imaging: Proximate Material Type Recognition in the Wild through Deep Learning of Spatial Surface Temperature Patterns

We introduce Deep Thermal Imaging, a new approach for close-range automatic recognition of materials to enhance the understanding of people and ubiquitous technologies of their proximal environment. Our approach uses a low-cost mobile thermal camera integrated into a smartphone to capture thermal textures. A deep neural network classifies these textures into material types. This approach works effectively without the need for ambient light sources or direct contact with materials. Furthermore, the use of a deep learning network removes the need to handcraft the set of features for different materials. We evaluated the performance of the system by training it to recognise 32 material types in both indoor and outdoor environments. Our approach produced recognition accuracies above 98% in 14,860 images of 15 indoor materials and above 89% in 26,584 images of 17 outdoor materials. We conclude by discussing its potentials for real-time use in HCI applications and future directions.Comment: Proceedings of the 2018 CHI Conference on Human Factors in Computing System

Thermographic Laplacian-pyramid filtering to enhance delamination detection in concrete structure

Despite decades of efforts using thermography to detect delamination in concrete decks, challenges still exist in removing environmental noise from thermal images. The performance of conventional temperature-contrast approaches can be significantly limited by environment-induced non-uniform temperature distribution across imaging spaces. Time-series based methodologies were found robust to spatial temperature non-uniformity but require the extended period to collect data. A new empirical image filtering method is introduced in this paper to enhance the delamination detection using blob detection method that originated from computer vision. The proposed method employs a Laplacian of Gaussian filter to achieve multi-scale detection of abnormal thermal patterns by delaminated areas. Results were compared with the state-of-the-art methods and benchmarked with time-series methods in the case of handling the non-uniform heat distribution issue. To further evaluate the performance of the method numerical simulations using transient heat transfer models were used to generate the 'theoretical' noise-free thermal images for comparison. Significant performance improvement was found compared to the conventional methods in both indoor and outdoor tests. This methodology proved to be capable to detect multi-size delamination using a single thermal image. It is robust to the non-uniform temperature distribution. The limitations were discussed to refine the applicability of the proposed procedure

HETEROGENEOUS MULTI-SENSOR FUSION FOR 2D AND 3D POSE ESTIMATION

Sensor fusion is a process in which data from different sensors is combined to acquire an output that cannot be obtained from individual sensors. This dissertation first considers a 2D image level real world problem from rail industry and proposes a novel solution using sensor fusion, then proceeds further to the more complicated 3D problem of multi sensor fusion for UAV pose estimation. One of the most important safety-related tasks in the rail industry is an early detection of defective rolling stock components. Railway wheels and wheel bearings are two components prone to damage due to their interactions with the brakes and railway track, which makes them a high priority when rail industry investigates improvements to current detection processes. The main contribution of this dissertation in this area is development of a computer vision method for automatically detecting the defective wheels that can potentially become a replacement for the current manual inspection procedure. The algorithm fuses images taken by wayside thermal and vision cameras and uses the outcome for the wheel defect detection. As a byproduct, the process will also include a method for detecting hot bearings from the same images. We evaluate our algorithm using simulated and real data images from UPRR in North America and it will be shown in this dissertation that using sensor fusion techniques the accuracy of the malfunction detection can be improved. After the 2D application, the more complicated 3D application is addressed. Precise, robust and consistent localization is an important subject in many areas of science such as vision-based control, path planning, and SLAM. Each of different sensors employed to estimate the pose have their strengths and weaknesses. Sensor fusion is a known approach that combines the data measured by different sensors to achieve a more accurate or complete pose estimation and to cope with sensor outages. In this dissertation, a new approach to 3D pose estimation for a UAV in an unknown GPS-denied environment is presented. The proposed algorithm fuses the data from an IMU, a camera, and a 2D LiDAR to achieve accurate localization. Among the employed sensors, LiDAR has not received proper attention in the past; mostly because a 2D LiDAR can only provide pose estimation in its scanning plane and thus it cannot obtain full pose estimation in a 3D environment. A novel method is introduced in this research that enables us to employ a 2D LiDAR to improve the full 3D pose estimation accuracy acquired from an IMU and a camera. To the best of our knowledge 2D LiDAR has never been employed for 3D localization without a prior map and it is shown in this dissertation that our method can significantly improve the precision of the localization algorithm. The proposed approach is evaluated and justified by simulation and real world experiments

Improving building energy modelling by applying advanced 3D surveying techniques on agri-food facilities

Food industry is the production sector with the highest energy consumption. In Europe, the energy used to produce food accounts for 26% of total energy consumption. Over 28% is used in industrial processes. Recently, European food companies have increased their efforts to make their production processes more sustainable, also by giving preference to the use of renewable energy sources. In Italy, the total energy consumption in agriculture and food sectors decreased between 2013 and 2014, passing from 16.79 to 13.3 Mtep. Since energy consumption in food industry is nearly twice the one in agriculture (8.57 and 4.73 Mtep, respectively), it is very important to improve energy efficiency and use green technologies in all the phases of food processing and conservation. In Italy, a recent law (Legislative Decree 102, 04/07/2014) has made energy-use diagnosis compulsory for all industrial concerns, particularly for those showing high consumption levels. In the case of food industry buildings, energy is mainly used for indoor microclimate control, which is needed to ensure workers' wellbeing and the most favourable conditions for food processing and conservation. To this end, it is important to have tools and methods allowing for easy, rapid and precise energy performance assessment of agri-food buildings. The accuracy of the results obtainable from the currently available computational models depends on the grade of detail and information used in constructional and geometric modelling. Moreover, this phase is probably the most critical and time-consuming in the energy diagnosis. In this context, fine surveying and advanced 3D geometric modelling procedures can facilitate building modelling and allow technicians and professionals in the agri-food sector to use highly efficient and accurate energy analysis and evaluation models. This paper proposes a dedicated model for energy performance assessment in agri-food buildings. It also shows that using advanced surveying techniques, such as a terrestrial laser scanner and an infrared camera, it is possible to create a three-dimensional parametric model, while, thanks to the heat flow meter Accepted paper measurement method, it is also possible to obtain a thermophysical model. This model allows assessing the energy performance of agri-food buildings in order to improve the indoor microclimate control and the conditions of food processing and conservation

Exergy-based Planning and Thermography-based Monitoring for energy efficient buildings - Progress Report (KIT Scientific Reports ; 7632)

Designing and monitoring energy efficiency of buildings is vital since they account for up to 40% of end-use energy. In this study, exergy analysis is investigated as a life cycle design tool to strike a balance between thermodynamic efficiency of energy conversion and economic and environmental costs of construction. Quantitative geo-referenced thermography is proposed for monitoring and quantitative assessment via continued simulation and parameter estimation during the operating phase

Computer Vision Applications for Autonomous Aerial Vehicles

Undoubtedly, unmanned aerial vehicles (UAVs) have experienced a great leap forward over the last decade. It is not surprising anymore to see a UAV being used to accomplish a certain task, which was previously carried out by humans or a former technology. The proliferation of special vision sensors, such as depth cameras, lidar sensors and thermal cameras, and major breakthroughs in computer vision and machine learning fields accelerated the advance of UAV research and technology. However, due to certain unique challenges imposed by UAVs, such as limited payload capacity, unreliable communication link with the ground stations and data safety, UAVs are compelled to perform many tasks on their onboard embedded processing units, which makes it difficult to readily implement the most advanced algorithms on UAVs. This thesis focuses on computer vision and machine learning applications for UAVs equipped with onboard embedded platforms, and presents algorithms that utilize data from multiple modalities. The presented work covers a broad spectrum of algorithms and applications for UAVs, such as indoor UAV perception, 3D understanding with deep learning, UAV localization, and structural inspection with UAVs. Visual guidance and scene understanding without relying on pre-installed tags or markers is the desired approach for fully autonomous navigation of UAVs in conjunction with the global positioning systems (GPS), or especially when GPS information is either unavailable or unreliable. Thus, semantic and geometric understanding of the surroundings become vital to utilize vision as guidance in the autonomous navigation pipelines. In this context, first, robust altitude measurement, safe landing zone detection and doorway detection methods are presented for autonomous UAVs operating indoors. These approaches are implemented on Google Project Tango platform, which is an embedded platform equipped with various sensors including a depth camera. Next, a modified capsule network for 3D object classification is presented with weight optimization so that the network can be fit and run on memory-constrained platforms. Then, a semantic segmentation method for 3D point clouds is developed for a more general visual perception on a UAV equipped with a 3D vision sensor. Next, this thesis presents algorithms for structural health monitoring applications involving UAVs. First, a 3D point cloud-based, drift-free and lightweight localization method is presented for depth camera-equipped UAVs that perform bridge inspection, where GPS signal is unreliable. Next, a thermal leakage detection algorithm is presented for detecting thermal anomalies on building envelopes using aerial thermography from UAVs. Then, building on our thermal anomaly identification expertise gained on the previous task, a novel performance anomaly identification metric (AIM) is presented for more reliable performance evaluation of thermal anomaly identification methods

Non-destructive techniques (NDT) for the diagnosis of heritage buildings: Traditional procedures and futures perspectives

It is estimated that EU cultural heritage (CH) buildings represent 30% of the total existing stock. Nevertheless, all actions in terms of refurbishment need a deep knowledge based on the diagnosis of the built quality. For this reason, the paper aims to provide a comprehensive review about the applicability of non-destructive techniques (NDT) and advanced modelling technologies for the diagnosis of heritage buildings. Considering a time span of two decades (2001–2021), a bibliometric analysis was performed, using data statistics and science mapping. Subsequently, the most relevant studies on this topic were evaluated for each technique. The main findings revealed that: (i) most of studies were conducted on Southern European countries; (ii) 36% of publications were journal papers and only 2% corresponded to reviews; (iii) “photogrammetry” and “laser applications” were identified as consolidated techniques for historic preservation, but they are only linked with HBIM and deep learning; (iv) a significant gap on quantitative NDT was detected and consequently, future researches should be performed to propose a common diagnosis protocol; (v) artificial neural networks have several barriers (i.e. data privacy, network security and quality of datasets). Hence, a holistic approach should be adopted by the European countries

Longitudinal thermal imaging for scalable non-residential HVAC and occupant behaviour characterization

This work presents a study on the characterization of the air-conditioning (AC) usage pattern of non-residential buildings from thermal images collected from an urban-scale infrared (IR) observatory. To achieve this first, an image processing scheme, for cleaning and extraction of the temperature time series from the thermal images is implemented. To test the accuracy of the thermal measurements using IR camera, the extracted temperature is compared against the ground truth surface temperature measurements. It is observed that the detrended thermal measurements match well with the ground truth surface temperature measurements. Subsequently, the operational pattern of the water-cooled systems and window AC units are extracted from the analysis of the thermal signature. It is observed that for the water-cooled system, the difference between the rate of change of the window and wall can be used to extract the operational pattern. While, in the case of the window AC units, wavelet transform of the AC unit temperature is used to extract the frequency and time domain information of the AC unit operation. The results of the analysis are compared against the indoor temperature sensors installed in the office spaces of the building. It is realized that the accuracy in the prediction of the operational pattern is highest between 8 pm to 10 am, and it reduces during the day because of solar radiation and high daytime temperature. Subsequently, a characterization study is conducted for eight window/split AC units from the thermal image collected during the nighttime. This forms one of the first studies on the operational behavior of HVAC systems for non-residential buildings using the longitudinal thermal imaging technique. The output from this study can be used to better understand the operational and occupant behavior, without requiring to deploy a large array of sensors in the building space