Visual Servoing NMPC Applied to UAVs for Photovoltaic Array Inspection

The photovoltaic (PV) industry is seeing a significant shift toward large-scale solar plants, where traditional inspection methods have proven to be time-consuming and costly. Currently, the predominant approach to PV inspection using unmanned aerial vehicles (UAVs) is based on photogrammetry. However, the photogrammetry approach presents limitations, such as an increased amount of useless data during flights, potential issues related to image resolution, and the detection process during high-altitude flights. In this work, we develop a visual servoing control system applied to a UAV with dynamic compensation using a nonlinear model predictive control (NMPC) capable of accurately tracking the middle of the underlying PV array at different frontal velocities and height constraints, ensuring the acquisition of detailed images during low-altitude flights. The visual servoing controller is based on the extraction of features using RGB-D images and the Kalman filter to estimate the edges of the PV arrays. Furthermore, this work demonstrates the proposal in both simulated and real-world environments using the commercial aerial vehicle (DJI Matrice 100), with the purpose of showcasing the results of the architecture. Our approach is available for the scientific community in: https://github.com/EPVelasco/VisualServoing_NMPCComment: This paper is under review at the journal "IEEE Robotics and Automation Letters

Light Unmanned Aerial Vehicles (UAVs) for cooperative inspection of PV plants

After a fast photovoltaic (PV) expansion in the past decade supported by many governments in Europe, in this postincentive era, one of the most significant open issues in the PV sector is to find appropriate inspection methods to evaluate real PV plant performance and failures. In this context, PV modules are surely the key components affecting the overall system performance; therefore, there is a main concern about the occurrence of any kind of failure in PV modules. This paper aims to propose a novel concept for monitoring PV plants by using light unmanned aerial vehicles (UAVs) or systems (UASs) during their operation and maintenance. The main objectives of this study are to explore and evaluate the use of different UAV technologies and to propose a reliable, cost-effective, and time-saving method for the inspection of PV plants. In this research, different UAVs were employed to inspect a PV array field. For this purpose, some thermal imaging cameras and a visual camera were chosen as monitoring tools to suitably scan PV modules. The first results show that the procedure of utilizing UAV was effective in the detection of different failures of PV modules. Moreover, such a process was much faster and cost effective than traditional methods

Desarrollo de geotecnologías aplicadas a la inspección y monitorización de entornos industriales

Tesis por compendio de publicaciones[ES]El desarrollo tecnológico de las últimas dos décadas ha supuesto un cambio radical que está llevando a un nuevo paradigma en el que se entremezclan el mundo físico y el digital. Estos cambios han influido enormemente en la sociedad, modificando las formas de comunicación, acceso a información, ocio, trabajo, etc. Asimismo, la industria ha adoptado estas tecnologías disruptivas, las cuales están contribuyendo a lograr un mayor control y automatización del proceso productivo. En el ámbito industrial, las tareas de mantenimiento son críticas para garantizar el correcto funcionamiento de una planta o instalación, ya que influyen directamente en la productividad y pueden suponer un elevado costo adicional. Las nuevas tecnologías están posibilitando la monitorización continua y a la inspección automatizada, proporcionando herramientas auxiliares a los inspectores que mejoran la detección de fallos y permiten anticipar y optimizar la planificación de las tareas de mantenimiento. Con el objetivo de desarrollar herramientas que aporten mejoras en las tareas de mantenimiento en industria, la presente tesis doctoral se basa en el estudio de como las geotecnologías pueden aportar soluciones óptimas en la monitorización e inspección. Debido a la gran variedad de entornos industriales, las herramientas de apoyo al mantenimiento deben adaptarse a cada caso en concreto. En este aspecto, y con el fin de demostrar la adaptabilidad de la geomática y las geotecnologías, se han estudiado instalaciones industriales de ámbitos muy diversos, como una sala de máquinas (escenario interior), plantas fotovoltaicas (escenario exterior) y soldaduras (interior y exterior). La escala de los escenarios objeto de estudio ha sido muy variada, desde las escalas más pequeñas, para el estudio de las soldaduras y la sala de máquinas, a las escalas más grandes, en los estudios de evolución de la vegetación y presencia de masas de agua en plantas fotovoltaicas. Las geotecnologías demuestran su versatilidad para trabajar a distintas escalas, con soluciones que permiten un gran detalle y precisión, como la fotogrametría de rango cercano y el sistema de escaneado portátil (Portable Mobile Mapping System - PMMS), y otras que pueden abarcar zonas más amplias del territorio, como es el caso de la teledetección o la fotogrametría con drones. Según lo expuesto anteriormente, el enfoque de la tesis ha sido el estudio de elementos o instalaciones industriales a diferentes escalas. En el primer caso se desarrolló una herramienta para el control de calidad externo de soldaduras utilizando fotogrametría de rango cercano y algoritmos para la detección automática de defectos. En el segundo caso se propuso el uso de un PMMS para optimizar la toma de datos en las tareas de inspección en instalaciones fluidomecánicas. En el tercer caso se utilizó la fotogrametría con drones y la combinación de imágenes RGB y térmicas con algoritmos de visión computacional para la detección de patologías en paneles fotovoltaicos. Finalmente, para la monitorización de la vegetación y la detección de masas de agua en el entorno de plantas fotovoltaicas, se empleó la teledetección mediante el cálculo de índices espectrales. [EN]The technological development of the last two decades has brought about a radical change that is leading to a new paradigm in which the physical and digital worlds are intertwined. These changes have had a great impact on society, modifying communication methods, access to information, leisure, work, etc. In addition, the industry has adopted these disruptive technologies, which are contributing to achieving greater control and automation of the production process. In the industrial sector, maintenance tasks are critical to ensuring the proper operation of a plant or facility, as they directly influence productivity and can involve high additional costs. New technologies are making continuous monitoring and automated inspection possible, providing auxiliary tools to inspectors that improve fault detection and allow for the anticipation and optimization of maintenance task planning. With the aim of developing tools that provide improvements in maintenance tasks in industry, this doctoral thesis is based on the study of how geotechnologies can provide optimal solutions in monitoring and inspection. Due to the great variety of industrial environments, maintenance support tools must adapt to each specific case. In this regard, and in order to demonstrate the adaptability of geomatics and geotechnologies, industrial installations from very diverse areas have been studied, such as a machine room (indoor scenario), photovoltaic plants (outdoor scenario), and welding (indoor and outdoor scenarios). The scale of the studied scenarios has been very varied, ranging from smaller scales for the study of welds and machine rooms, to larger scales in the studies of vegetation evolution and the presence of bodies of water in photovoltaic plants. Geotechnologies demonstrate their versatility to work at different scales, with solutions that allow for great detail and precision, such as close-range photogrammetry and the Portable Mobile Mapping System (PMMS), as well as others that can cover larger areas of the territory, such as remote sensing or photogrammetry with drones. The focus of the thesis has been the study of industrial elements or installations at different scales. In the first case, a tool was developed for external quality control of welding, using close-range photogrammetry and algorithms for automatic defect detection. In the second case, the use of a PMMS is proposed to optimize data collection in fluid-mechanical installation inspection tasks. In the third case, drone photogrammetry and the combination of RGB and thermal images with computer vision algorithms were used for the detection of pathologies in photovoltaic panels. Finally, for the monitoring of vegetation and the detection of water masses in the environment of photovoltaic plants, remote sensing was employed through the calculation of spectral indices

A Photovoltaic Module Diagnostic Setup for Lock-in Electroluminescence Imaging

Electroluminescence (EL) imaging and infrared (IRT) thermography techniques have become indispensable tools in recent years for health diagnostic of photovoltaic modules in solar industry application. We propose a diagnostic setup, which performs lock-in EL for accurate analysis of different types of faults occurring in a solar module. The setup is built around a high-speed SWIR camera, which can acquire images at very short integration time (1μs) and high frame rate (301 fps). In addition, a state-of-the-art imaging chamber allows for introducing controlled levels of ambient light noise for developing new light noise removal methods, rotation of panel frame in 3 axes plane for developing perspective distortion correction techniques. The paper also gives an insight of different system and communication delays that affects the performance of overall EL lock-in imaging system integration. The purpose of the diagnostic setup is to support research in PV failure quantification through EL imaging, which can also be useful for aerial drone imaging of PV plants.</p

Development of outdoor luminescence imaging for drone-based PV array inspection

This work has the goal to perform outdoor defect detection imaging that will be used in a fast, accurate and automatic drone-based survey system for PV power plants. The imaging development focuses on techniques that do not require electrical contact, permitting automatic drone inspections to be perform quicker and with less manpower. The final inspection method will combine several techniques such as, infrared (IR), electroluminescence (EL), photoluminescence (PL), and visual imaging. Solar plant inspection in the future can be restricted only by imaging speed requirements, allowing an entire new perspective in large-scale PV inspection

A Comparative Evaluation of Deep Learning Techniques for Photovoltaic Panel Detection From Aerial Images

Solar energy production has significantly increased in recent years in the European Union (EU), accounting for 12% of the total in 2022. The growth in solar energy production can be attributed to the increasing adoption of solar photovoltaic (PV) panels, which have become cost-effective and efficient means of energy production, supported by government policies and incentives. The maturity of solar technologies has also led to a decrease in the cost of solar energy, making it more competitive with other energy sources. As a result, there is a growing need for efficient methods for detecting and mapping the locations of PV panels. Automated detection can in fact save time and resources compared to manual inspection. Moreover, the resulting information can also be used by governments, environmental agencies and other companies to track the adoption of renewable sources or to optimize energy distribution across the grid. However, building effective models to support the automated detection and mapping of solar photovoltaic (PV) panels presents several challenges, including the availability of high-resolution aerial imagery and high-quality, manually-verified labels and annotations. In this study, we address these challenges by first constructing a dataset of PV panels using very-high-resolution (VHR) aerial imagery, specifically focusing on the region of Piedmont in Italy. The dataset comprises 105 large-scale images, providing more than 9,000 accurate and detailed manual annotations, including additional attributes such as the PV panel category. We first conduct a comprehensive evaluation benchmark on the newly constructed dataset, adopting various well-established deep-learning techniques. Specifically, we experiment with instance and semantic segmentation approaches, such as Rotated Faster RCNN and Unet, comparing strengths and weaknesses on the task at hand. Second, we apply ad-hoc modifications to address the specific issues of this task, such as the wide range of scales of the installations and the sparsity of the annotations, considerably improving upon the baseline results. Last, we introduce a robust and efficient post-processing polygonization algorithm that is tailored to PV panels. This algorithm converts the rough raster predictions into cleaner and more precise polygons for practical use. Our benchmark evaluation shows that both semantic and instance segmentation techniques can be effective for detecting and mapping PV panels. Instance segmentation techniques are well-suited for estimating the localization of panels, while semantic solutions excel at surface delineation. We also demonstrate the effectiveness of our ad-hoc solutions and post-processing algorithm, which can provide an improvement up to +10% on the final scores, and can accurately convert coarse raster predictions into usable polygons

Drone-based non-destructive inspection of industrial sites: a review and case studies

Using aerial platforms for Non-Destructive Inspection (NDI) of large and complex structures is a growing field of interest in various industries. Infrastructures such as: buildings, bridges, oil and gas, etc. refineries require regular and extensive inspections. The inspection reports are used to plan and perform required maintenance, ensuring their structural health and the safety of the workers. However, performing these inspections can be challenging due to the size of the facility, the lack of easy access, the health risks for the inspectors, or several other reasons, which has convinced companies to invest more in drones as an alternative solution to overcome these challenges. The autonomous nature of drones can assist companies in reducing inspection time and cost. Moreover, the employment of drones can lower the number of required personnel for inspection and can increase personnel safety. Finally, drones can provide a safe and reliable solution for inspecting hard-to-reach or hazardous areas. Despite the recent developments in drone-based NDI to reliably detect defects, several limitations and challenges still need to be addressed. In this paper, a brief review of the history of unmanned aerial vehicles, along with a comprehensive review of studies focused on UAV-based NDI of industrial and commercial facilities, are provided. Moreover, the benefits of using drones in inspections as an alternative to conventional methods are discussed, along with the challenges and open problems of employing drones in industrial inspections, are explored. Finally, some of our case studies conducted in different industrial fields in the field of Non-Destructive Inspection are presented

Mapping urban surface materials using imaging spectroscopy data

Die Kartierung der städtische Oberflächenmaterialien ist aufgrund der komplexen räumlichen Muster eine Herausforderung. Daten von bildgebenden Spektrometern können hierbei durch die feine und kontinuierliche Abtastung des elektromagnetischen Spektrums detaillierte spektrale Merkmale von Oberflächenmaterialien erkennen, was mit multispektralen oder RGB-Bildern nicht mit der gleichen Genauigkeit erreicht werden kann. Bislang wurden in zahlreichen Studien zur Kartierung von städtischen Oberflächenmaterialien Daten von flugzeuggestützten abbildenden Spektrometern mit hoher räumlicher Auflösung verwendet, die ihr Potenzial unter Beweis stellen und gute Ergebnisse liefern. Im Vergleich zu diesen Sensoren haben weltraumgestützte abbildende Spektrometer eine regionale oder globale Abdeckung, eine hohe Wiederholbarkeit und vermeiden teure, zeit- und arbeitsaufwändige Flugkampagnen. Allerdings liegt die räumliche Auflösung der aktuellen weltraumgestützten abbildenden Spektroskopiedaten bei etwa 30 m, was zu einem Mischpixelproblem führt, welches mit herkömmlichen Kartierungsansätzen nur schwer zu bewältigen ist. Das Hauptziel dieser Studie ist die Kartierung städtischer Materialien mit bildgebenden Spektroskopiedaten in verschiedenen Maßstäben und die gleichzeitige Nutzung des Informationsgehalts dieser Daten, um die chemischen und physikalischen Eigenschaften von Oberflächenmaterialien zu erfassen sowie das Mischpixelproblem zu berücksichtigen. Konkret zielt diese Arbeit darauf ab, (1) photovoltaische Solarmodule mit Hilfe von luftgestützten bildgebenden Spektroskopiedaten auf der Grundlage ihrer spektralen Merkmale zu kartieren; (2) die Robustheit der Stichprobe von städtischen Materialgradienten zu untersuchen; (3) die Übertragbarkeit von städtischen Materialgradienten auf andere Gebiete zu analysieren.Mapping urban surface materials is challenging due to the complex spatial patterns. Data from imaging spectrometers can identify detailed spectral features of surface materials through the fine and continuous sampling of the electromagnetic spectrum, which cannot be achieved with the same accuracy using multispectral or RGB images. To date, numerous studies in urban surface material mapping have been using data from airborne imaging spectrometers with high spatial resolution, demonstrating the potential and providing good results. Compared to these sensors, spaceborne imaging spectrometers have regional or global coverage, high repeatability, and avoid expensive, time-consuming, and labor-intensive flight campaigns. However, the spatial resolution of current spaceborne imaging spectroscopy data (also known as hyperspectral data) is about 30 m, resulting in a mixed pixel problem that is challenging to handle with conventional mapping approaches. The main objective of this study is to perform urban surface material mapping with imaging spectroscopy data at different spatial scales, simultaneously explore the information content of these data to detect the chemical and physical properties of surface materials, and take the mixed-pixel problem into account. Specifically, this thesis aims to (1) map solar photovoltaic modules using airborne imaging spectroscopy data based on their spectral features; (2) investigate the sampling robustness of urban material gradients; (3) analyze the area transferability of urban material gradients