14 research outputs found

    RGB-D And Thermal Sensor Fusion: A Systematic Literature Review

    Full text link
    In the last decade, the computer vision field has seen significant progress in multimodal data fusion and learning, where multiple sensors, including depth, infrared, and visual, are used to capture the environment across diverse spectral ranges. Despite these advancements, there has been no systematic and comprehensive evaluation of fusing RGB-D and thermal modalities to date. While autonomous driving using LiDAR, radar, RGB, and other sensors has garnered substantial research interest, along with the fusion of RGB and depth modalities, the integration of thermal cameras and, specifically, the fusion of RGB-D and thermal data, has received comparatively less attention. This might be partly due to the limited number of publicly available datasets for such applications. This paper provides a comprehensive review of both, state-of-the-art and traditional methods used in fusing RGB-D and thermal camera data for various applications, such as site inspection, human tracking, fault detection, and others. The reviewed literature has been categorised into technical areas, such as 3D reconstruction, segmentation, object detection, available datasets, and other related topics. Following a brief introduction and an overview of the methodology, the study delves into calibration and registration techniques, then examines thermal visualisation and 3D reconstruction, before discussing the application of classic feature-based techniques as well as modern deep learning approaches. The paper concludes with a discourse on current limitations and potential future research directions. It is hoped that this survey will serve as a valuable reference for researchers looking to familiarise themselves with the latest advancements and contribute to the RGB-DT research field.Comment: 33 pages, 20 figure

    Herramienta software para la calibración extrínseca de cámaras infrarrojas y RGBD aplicada a inspección termográfica

    Get PDF
    Context:  Thermographic inspections are currently used to assess energy efficiency in electrical equipment and civil structures or to detect failures in cooling systems and electrical or electronic devices. However, thermal images lack texture details, which does not allow for a precise identification of the geometry of the scene or the objects in it. Method: In this work, the development of the software tool called DepTherm is described. This tool allows performing intrinsic and extrinsic calibration between infrared and RGBD cameras in order to fuse thermal, RGB, and RGBD images, as well as to record thermal and depth data. Additional features include user management, a visualization GUI for all three types of images, database storage, and report generation. Results: In addition to the integration tests performed to validate the functionality of DepTherm, two quantitative tests were conducted in order to evaluate its accuracy. A maximum re-projection error of 1,47±0,64 pixels was found, and the maximum mean error in registering an 11 cm side cube was 4,15 mm. Conclusions: The features of the DepTherm software tool are focused on facilitating thermographic inspections by capturing 3D scene models with thermal data.Contexto: Las inspecciones termográficas se utilizan en la actualidad para evaluar la eficiencia energética de equipos eléctricos y estructuras civiles o para detectar fallas en sistemas de enfriamiento y dispositivos eléctricos o electrónicos. Sin embargo, las imágenes térmicas carecen de detalles de textura, lo cual no permite identificar con precisión la geometría de la escena ni los objetos en ella. Método: En este trabajo se describe el desarrollo de la herramienta de software DepTherm, la cual permite realizar calibraciones intrínsecas y extrínsecas entre cámaras infrarrojas y RGBD para fusionar imágenes térmicas, RGB y RGBD, así como para registrar datos térmicos y de profundidad. Funcionalidades adicionales incluyen el manejo de usuarios, una GUI para visualización de los tres tipos de imágenes, el almacenamiento en una base de datos y la generación de reportes. Resultados: Además de las pruebas de integración para validar la funcionalidad de DepTherm, se realizaron dos pruebas cuantitativas para evaluar su precisión. Se encontró un error máximo de reproyección de 1,47±0,64 pixeles, mientras que el registro de un cubo con 11 cm de lado tuvo un error promedio máximo de 4,147 mm. Conclusiones: Las funcionalidades de la herramienta software DepTherm están enfocadas en facilitar las inspecciones termográficas capturando modelos 3D de las escenas con información térmica

    Laboratorij za autonomne sustave i mobilnu robotiku

    Get PDF
    Laboratorij za autonomne sustave i mobilnu robotiku (LAMOR) istraživački je laboratorij koji djeluje u okviru Zavoda za automatiku i računalno inženjerstvo Fakulteta elektrotehnike i računarstva Sveučilišta u Zagrebu. LAMOR je jedan od vodećih laboratorija u Republici Hrvatskoj u području robotike, a svoje je istraživačko djelovanje usmjerio na temeljna istraživanja algoritama upravljanja, estimacije i umjetne inteligencije s primjenom u razvoju sustava autonomije mobilnih robota i vozila u nepoznatim i dinamičkim okruženjima te sustava djelotvorne i sigurne interakcije autonomnih mobilnih robota i ljudi. U radu je opisana uspostava laboratorija i prikazan je njegov 20-godišnji razvoj, a potom su predstavljene osnovne informacije o njegovoj istraživačkoj djelatnosti, najznačajnijim znanstvenim postignućima, najvažnijim istraživačkim projektima, međunarodnoj suradnji te doprinosu razvoju znanosti i gospodarstva u Republici Hrvatskoj

    Laboratorij za autonomne sustave i mobilnu robotiku

    Get PDF
    Laboratorij za autonomne sustave i mobilnu robotiku (LAMOR) istraživački je laboratorij koji djeluje u okviru Zavoda za automatiku i računalno inženjerstvo Fakulteta elektrotehnike i računarstva Sveučilišta u Zagrebu. LAMOR je jedan od vodećih laboratorija u Republici Hrvatskoj u području robotike, a svoje je istraživačko djelovanje usmjerio na temeljna istraživanja algoritama upravljanja, estimacije i umjetne inteligencije s primjenom u razvoju sustava autonomije mobilnih robota i vozila u nepoznatim i dinamičkim okruženjima te sustava djelotvorne i sigurne interakcije autonomnih mobilnih robota i ljudi. U radu je opisana uspostava laboratorija i prikazan je njegov 20-godišnji razvoj, a potom su predstavljene osnovne informacije o njegovoj istraživačkoj djelatnosti, najznačajnijim znanstvenim postignućima, najvažnijim istraživačkim projektima, međunarodnoj suradnji te doprinosu razvoju znanosti i gospodarstva u Republici Hrvatskoj

    Three-dimensional eddy current pulsed thermography and its applications

    Get PDF
    Ph. D. Thesis.The measurement and quantification of defects is a challenge for Non-DestructiveTesting and Evaluation (NDT&E). Such challenges include the precise localisation and detection of surface and sub-surface defects, as well as the quantification of such defects. This work first reports a three-dimensional (3D) Eddy Current Pulsed Thermography (ECPT) system via integration with an RGB-D camera. Then, various quantitative measurements and analyses of defects are carried out based on the 3D ECPT system. The ECPT system at Newcastle University has been prooven to be an effective nondestructive testing (NDT) method in surface and sub-surface detection over the past few years. Based on the different numerical or analytical models, it has achieved precise defect detection on the rail tracks, wind turbines, carbon fibre reinforced plastic (CFRP) and so on. The ECPT system has the advantage of fast inspection and a large lift-off range. However, it involves a trade-off between detectable defect size and inspection area compared with other NDT methods. In addition, there are challenges of defect detection in a complex structure. Thus, the quantification of defects gives a higher requirement of the measurement the object geometry information. Furthermore, the analysis of thermal diffusion requires a precise 3D model. For this reason, a 3D ECPT system is proposed that adds each heat pixel with an exact X-Y-Z coordinate. In this work, first, the 3D ECPT system is built. A feature-based automatic calibration of the infrared camera and the RGB-D camera is proposed. Second, the software platform is built. A fast 3D visualization is completed with multi-threading technology and the Point Cloud Library. Lastly, various studies of defect localization, quantification and thermal tomography reconstruction are carried ou

    Lighting in the third dimension : laser scanning as an architectural survey and representation method

    Get PDF
    This paper proposes tridimensional (3D) laser scanning to architects and lighting designers as a lighting enquiry and visualization method for existing built environments. The method constitutes a complement to existing lighting methods by responding to limitations of photometric measurements, computer simulation and HDR imagery in surveying and visualizing light in actual buildings. The research explores advantages and limitations of 3D laser scanning in a case study addressing a vast, geometrically complex and fragmented naturally and artificially lit space. Lighting patterns and geometry of the case study are captured with a 3D laser scanner through a series of four scans. A single 3D model of the entire space is produced from the aligned and fused scans. Lighting distribution patterns are showcased in relation to the materiality, geometry and position of windows, walls, lighting fixtures and day lighting sources. Results and presented through images similar to architectural presentation drawings. More specifically, the lighting distribution patterns are illustrated in a floor plan, a reflected ceiling plan, an axonometry and a cross-section. The point cloud model of the case study is also generated into a video format representing the entire building as well as different viewpoints. The study shows that the proposed method provides powerful visualization results due to the unlimited number of images that can be generated from a point cloud and facilitates understanding of existing lighting conditions in spaces

    Towards Airborne Thermography via Low-Cost Thermopile Infrared Sensors

    Get PDF
    International audienceThis paper presents a novel tool capable of collecting thermal signatures inside a building by using low-cost IR temperature sensors mounted on-board an aerial platform. The proposed system aims to facilitate the detection of heat loss inside buildings, which is a key aspect for improving energy efficiency in large commercial or industrial buildings. Current detection systems usually require manual labor as well as the use of expensive instrumentation. The proposed system on the other hand, relies on the use of a small unmanned aerial vehicle carrying low-cost thermopile IR sensors. Moreover, the system delivers a fast temperature sensing scheme and it provides coverage to inaccessible areas, thus overcoming the limitations of current mobile platforms which use ground robots. Different experiments were carried out in order to assess the behavior of the sensors as well as to validate the full system. Moreover, the hypothesis that thermopile IR sensors can be used to track temperature signature on-the-fly is validated experimentally with the use of the proposed system over different targets

    Relevés lumineux tridimensionnels en architecture

    Get PDF
    Ce mémoire de maîtrise propose la numérisation laser tridimensionnelle LiDARcomme nouvelle méthode d'étude et de visualisation de l'éclairage naturel dans des environnements réels pour architectes et designers. Elle constitue un complément aux méthodes d'éclairage actuelles car elle répond aux limites de la méthode de mesure de l'éclairement, de la simulation numérique et de l'imagerie à haute dynamique(HDR)en ce qui concerne le relevé et la représentation des patterns lumineux. Il présente une étude de cas pour déterminer les avantages et les limites de la numérisation laser 3D dans une vaste cafétéria à éclairage naturel et artificiel, vaste, géométriquement complexe et fragmentée. Les patterns lumineuxet la géométrie del'espace sont capturés par un appareil à balayage laser 3D à travers une série de quatre numérisations. Les numérisations sont alignées et fusionnées pour former un seul modèle 3D de l'espace entier. Les patterns lumineuxsont présentés en relation avec la matérialité, la géométrie et la position des fenêtres, des murs, des appareils d'éclairage et des sources d'éclairage et présentés sous forme d'images semblables à des dessins de présentation architecturaux. Plus précisément, les patterns lumineuxsont illustrés dans un plan d'étage, un plan de plafond réfléchi, une axonométrie et une coupe transversale. La méthode fournit des résultats de visualisation percutants. Elle facilite leur compréhension des patterns lumineux, car un nombre illimité d'images peut être généré à partir d'un nuage de points. L'exactitude de la méthode de relevé des espaces éclairés naturellement estégalementvérifié pour desespacesrelevésen une et plusieurs numérisations en comparant les patterns lumineux des imageries HDR et des nuages de points. De plus, le mémoire explore le potentiel de la numérisation laser tridimensionnelle comme méthode pour simuler de nouvelles ambiances lumineuses dans des espaces existants.This master thesisproposesLiDARtridimensional laser scanning as a new daylighting enquiry and visualization method for real built environments for architects and designers. It constitutes a complement to actual lighting methods because it responds to the limitations of the illuminance measuring method, computer simulation and high dynamic rangeimagery concerning the survey and representation of lighting patterns. It presents a case study to determine the advantages and limitations of 3D laser scanning in a vast, geometrically complex and fragmented naturally and artificially lit cafeteria. Lighting patterns and the geometry of the space are captured with a 3D laser scanner through a series of four scans. The scans are aligned and fused to form a single 3D model of the entire space. The lighting distribution patterns are showcased in relation to the materiality, geometry and position of windows, walls, lighting fixtures and the lighting sources and presented through images similar to architectural presentation drawings. More precisely, the lighting distribution patterns are illustrated in a floor plan, a reflected ceiling plan, an axonometry and a cross-section. The method provides powerful visualization results and facilitates their understanding as an unlimited number of images can be generated from a point cloud.The precision of the method for surveying daylit environments surveyed through one and several scans is also verified by comparing lighting patterns between HDR and point cloud imageries. Moreover, it explores tridimensional laser scanning as a method for rendering new lighting ambiances in existing spaces

    Designing and Evaluating Next-Generation Thermographic Systems to Support Residential Energy Audits

    Get PDF
    Buildings account for 41% of primary energy consumption in the United States—more than any other sector—and contribute to an increasing portion of carbon dioxide emissions (33% in 1980 vs. 40% in 2009). To help address this problem, the U.S. Department of Energy recommends conducting energy audits to identify sources of inefficiencies that contribute to rising energy use. One effective technique used during energy audits is thermography. Thermographic-based energy auditing activities involve the use of thermal cameras to identify, diagnose, and document energy efficiency issues in the built environment that are visible as anomalous patterns of electromagnetic radiation. These patterns may indicate locations of air leakages, areas of missing insulation, or moisture issues in the built environment. Sensor improvements and falling costs have increased the popularity of this auditing technique, but its effectiveness is often mediated by the training and experience of the auditor. Moreover, given the increasing availability of commodity thermal cameras and the potential for pervasive thermographic scanning in the built environment, there is a surprising lack of understanding about people’s perceptions of this sensing technology and the challenges encountered by an increasingly diverse population of end-users. Finally, there are few specialized tools and methods to support the auditing activities of end-users. To help address these issues, my work focuses on three areas: (i) formative studies to understand and characterize current building thermography practices, benefits, and challenges, (ii) human-centered explorations into the role of automation and the potential of pervasive thermographic scanning in the built environment, and (iii) evaluations of novel, interactive building thermography systems. This dissertation presents a set of studies that qualitatively characterizes building thermography practitioners, explores prototypes of novel thermographic systems at varying fidelity, and synthesizes findings from several field deployments. This dissertation contributes to the fields of sustainability, computer science, and HCI through: (i) characterizations of the end-users of thermography, (ii) critical feedback on proposed automated thermographic solutions, (iii) the design and evaluation of a novel longitudinal thermography system designed to augment the data collection and analysis activities of end-users, and (iv) design recommendations for future thermographic systems

    Advancing Embedded and Extrinsic Solutions for Optimal Control and Efficiency of Energy Systems in Buildings

    Get PDF
    Buildings account for approximately 40% of all U.S. energy usage and carbon emissions. Reducing energy usage and improving efficiency in buildings has the potential for significant environmental and economic impacts. To do so, reoccurring identification of hardware and operational opportunities is needed to maintain building efficiency. Additionally, the development of controls that continually operate building systems and equipment at energy optimal conditions is required. This dissertation provides contributions to both of the aforementioned areas, which can be divided into two distinct portions. The first presents the framework for the development of an automated energy audit process, termed Autonomous Robotic Assessments of Energy (AuRAE). The automation of energy audits would decrease the cost of audits to customers, reduce the time auditors need to invest in an audit, and provide repeatable audit processes with enhanced data collection. In this framework of AuRAE, novel, audit-centric navigational strategies are presented that enable the complete exploration of a previously unknown space in a building while identifying and navigating to objects of interest in real-time as well as navigation around external building perimeters. Simulations of the navigational strategies show success in a variety of building layouts and size of objects of interest. Additionally, prototypes of robotic audit capabilities are demonstrated in the form of a lighting identification and analysis package on a ground vehicle and an environmental baseline measurement package on an aerial vehicle. The second portion presents the development and simulation of two advanced economic building energy controllers: one utilizes steady-state relationships for optimizing control setpoints while the other is an economic MPC method using dynamic models to optimize the same control setpoints. Both control methods balance the minimization of utility cost from energy usage with the cost of lost productivity due to occupant discomfort, differing from standard building optimal control that generally addresses occupant comfort through setpoint limits or comfort measure constraints. This is accomplished through the development of component-level economic objective functions for each subsystem in the modeled building. The results show that utility cost and the cost of occupant productivity from optimal comfort can be successfully balanced, and even improved over current control methods. The relative magnitude of the cost of lost productivity is shown to be significantly higher than the cost of utilities, suggesting that building operators, technicians, and researchers should make maintaining occupant comfort a top priority to achieve the greatest economic savings. Furthermore, the results demonstrate that by using steady-state predictions, the majority of the performance gains produced with a fully dynamic MPC solution can be recovered
    corecore