Small unmanned airborne systems to support oil and gas pipeline monitoring and mapping

Acknowledgments We thank Johan Havelaar, Aeryon Labs Inc., AeronVironment Inc. and Aeronautics Inc. for kindly permitting the use of materials in Fig. 1.Peer reviewedPublisher PD

A Distributed Approach for Collision Avoidance between Multirotor UAVs Following Planned Missions

[EN] As the number of potential applications for Unmanned Aerial Vehicles (UAVs) keeps rising steadily, the chances that these devices get close to each other during their flights also increases, causing concerns regarding potential collisions. This paper proposed the Mission Based Collision Avoidance Protocol (MBCAP), a novel UAV collision avoidance protocol applicable to all types of multicopters flying autonomously. It relies on wireless communications in order to detect nearby UAVs, and to negotiate the procedure to avoid any potential collision. Experimental and simulation results demonstrated the validity and effectiveness of the proposed solution, which typically introduces a small overhead in the range of 15 to 42 s for each risky situation successfully handled.This work was partially supported by the "Ministerio de Ciencia, Innovacion y Universidades, Programa Estatal de Investigacion, Desarrollo e Innovacion Orientada a los Retos de la Sociedad, Proyectos I+D+I 2018", Spain, under Grant RTI2018-096384-B-I00, and the Universitat Politecnica de Valencia (UPV) under grant number FPI-2017-S1 for the training of PhD researchers.Fabra Collado, FJ.; Zamora-Mero, WJ.; Sangüesa-Escorihuela, JA.; Tavares De Araujo Cesariny Calafate, CM.; Cano, J.; Manzoni, P. (2019). A Distributed Approach for Collision Avoidance between Multirotor UAVs Following Planned Missions. Sensors. 19(10):1-25. https://doi.org/10.3390/s19102404S1251910Mohamed, N., Al-Jaroodi, J., Jawhar, I., Idries, A., & Mohammed, F. (2020). Unmanned aerial vehicles applications in future smart cities. Technological Forecasting and Social Change, 153, 119293. doi:10.1016/j.techfore.2018.05.004SESAR Joint Undertakinghttps://www.sesarju.eu/Fabra, F., T. Calafate, C., Cano, J.-C., & Manzoni, P. (2018). MBCAP: Mission Based Collision Avoidance Protocol for UAVs. 2018 IEEE 32nd International Conference on Advanced Information Networking and Applications (AINA). doi:10.1109/aina.2018.00090Drone Collision Avoidancehttps://create.arduino.cc/projecthub/anshulsingh163/drone-collision-avoidance-system-0b6002Liu, Z., & Foina, A. G. (2016). Feature article: an autonomous quadrotor avoiding a helicopter in low-altitude flights. IEEE Aerospace and Electronic Systems Magazine, 31(9), 30-39. doi:10.1109/maes.2016.150131Xiang, J., Liu, Y., & Luo, Z. (2016). Flight safety measurements of UAVs in congested airspace. Chinese Journal of Aeronautics, 29(5), 1355-1366. doi:10.1016/j.cja.2016.08.017Lin, Q., Wang, X., & Wang, Y. (2018). Cooperative Formation and Obstacle Avoidance Algorithm for Multi-UAV System in 3D Environment. 2018 37th Chinese Control Conference (CCC). doi:10.23919/chicc.2018.8483113Zhou, X., Yu, X., & Peng, X. (2019). UAV Collision Avoidance Based on Varying Cells Strategy. IEEE Transactions on Aerospace and Electronic Systems, 55(4), 1743-1755. doi:10.1109/taes.2018.2875556Kim, H., & Ben-Othman, J. (2018). A Collision-Free Surveillance System Using Smart UAVs in Multi Domain IoT. IEEE Communications Letters, 22(12), 2587-2590. doi:10.1109/lcomm.2018.2875477Wang, M., Voos, H., & Su, D. (2018). Robust Online Obstacle Detection and Tracking for Collision-Free Navigation of Multirotor UAVs in Complex Environments. 2018 15th International Conference on Control, Automation, Robotics and Vision (ICARCV). doi:10.1109/icarcv.2018.8581330Ma, L. (2018). Cooperative Target Tracking using a Fleet of UAVs with Collision and Obstacle Avoidance. 2018 22nd International Conference on System Theory, Control and Computing (ICSTCC). doi:10.1109/icstcc.2018.8540717Chen, P.-H., & Lee, C.-Y. (2018). UAVNet: An Efficient Obstacel Detection Model for UAV with Autonomous Flight. 2018 International Conference on Intelligent Autonomous Systems (ICoIAS). doi:10.1109/icoias.2018.8494201Fabra, F., Calafate, C. T., Cano, J. C., & Manzoni, P. (2018). ArduSim: Accurate and real-time multicopter simulation. Simulation Modelling Practice and Theory, 87, 170-190. doi:10.1016/j.simpat.2018.06.009Accurate and real-time multi-UAV simulationhttps://bitbucket.org/frafabco/ardusim/src/master/MAVLink Micro Air Vehicle Communication Protocolhttp://qgroundcontrol.org/mavlink/startGorelick, N., Hancher, M., Dixon, M., Ilyushchenko, S., Thau, D., & Moore, R. (2017). Google Earth Engine: Planetary-scale geospatial analysis for everyone. Remote Sensing of Environment, 202, 18-27. doi:10.1016/j.rse.2017.06.031NS-2 The Network Simulatorhttp://nsnam.sourceforge.net/wiki/index.php/Main_PageOMNeT++ Discrete Event Simulatorhttps://omnetpp.org/Quaternium, Home of the Longest Flight Time Hybrid Dronehttp://www.quaternium.com/Gauss-Markov Mobilityhttps://doc.omnetpp.org/inet/api-current/neddoc/inet.mobility.single.GaussMarkovMobility.htmlFerrera, E., Alcántara, A., Capitán, J., Castaño, A., Marrón, P., & Ollero, A. (2018). Decentralized 3D Collision Avoidance for Multiple UAVs in Outdoor Environments. Sensors, 18(12), 4101. doi:10.3390/s1812410

A Co-optimal Coverage Path Planning Method for Aerial Scanning of Complex Structures

The utilization of unmanned aerial vehicles (UAVs) in survey and inspection of civil infrastructure has been growing rapidly. However, computationally efficient solvers that find optimal flight paths while ensuring high-quality data acquisition of the complete 3D structure remains a difficult problem. Existing solvers typically prioritize efficient flight paths, or coverage, or reducing computational complexity of the algorithm – but these objectives are not co-optimized holistically. In this work we introduce a co-optimal coverage path planning (CCPP) method that simultaneously co-optimizes the UAV path, the quality of the captured images, and reducing computational complexity of the solver all while adhering to safety and inspection requirements. The result is a highly parallelizable algorithm that produces more efficient paths where quality of the useful image data is improved. The path optimization algorithm utilizes a particle swarm optimization (PSO) framework which iteratively optimizes the coverage paths without needing to discretize the motion space or simplify the sensing models as is done in similar methods. The core of the method consists of a cost function that measures both the quality and efficiency of a coverage inspection path, and a greedy heuristic for the optimization enhancement by aggressively exploring the viewpoints search spaces. To assess the proposed method, a coverage path quality evaluation method is also presented in this research, which can be utilized as the benchmark for assessing other CPP methods for structural inspection purpose. The effectiveness of the proposed method is demonstrated by comparing the quality and efficiency of the proposed approach with the state-of-art through both synthetic and real-world scenes. The experiments show that our method enables significant performance improvement in coverage inspection quality while preserving the path efficiency on different test geometries

A New Methodology for Bridge Inspections in Linear Infrastructures from Optical Images and HD Videos Obtained by UAV

Many bridges and other structures worldwide present a lack of maintenance or a need for rehabilitation. The first step in the rehabilitation process is to perform a bridge inspection to know the bridge′s current state. Routine bridge inspections are usually based only on visual recognition. In this paper, a methodology for bridge inspections in communication routes using images acquired by unmanned aerial vehicle (UAV) flights is proposed. This provides access to the upper parts of the structure safely and without traffic disruptions. Then, a standardized and systematized novel image acquisition protocol is applied for data acquisition. Afterwards, the images are studied by civil engineers for damage identification and description. Then, specific structural inspection forms are completed using the acquired information. Recommendations about the need of new and more detailed inspections should be included at this stage when needed. The suggested methodology was tested on two railway bridges in France. Image acquisition of these structures was performed using an UAV for its ability to provide an expert assessment of the damage level. The main advantage of this method is that it makes it possible to safely accurately identify diverse damages in structures without the need for a specialised engineer to go to the site. Moreover, the videos can be watched by as many engineers as needed with no personal movement. The main objective of this work is to describe the systematized methodology for the development of bridge inspection tasks using a UAV system. According to this proposal, the in situ inspection by a specialised engineer is replaced by images and videos obtained from an UAV flight by a trained flight operator. To this aim, a systematized image/videos acquisition method is defined for the study of the morphology and typology of the structural elements of the inspected bridges. Additionally, specific inspection forms are proposed for every type of structural element. The recorded information will allow structural engineers to perform a postanalysis of the damage affecting the bridges and to evaluate the subsequent recommendations.This research was funded by the Shift2Rail Joint Undertaking under the European Union’s Horizon 2020 research and innovation program, with grant agreement No 777630, project MOMIT, “Multiscale Observation and Monitoring of railway Infrastructure Threats”

Flight coordination solutions for multirotor unmanned aerial vehicles

[EN] As the popularity and the number of Unmanned Aerial Vehicles (UAVs) increases, new protocols are needed to coordinate them when flying without direct human control, and to avoid that these UAVs collide with each other. Testing such novel protocols on real UAVs is a complex procedure that requires investing much time, money and research efforts. Hence, it becomes necessary to first test the developed solutions using simulation. Unfortunately, existing tools present significant limitations: some of them only simulate accurately the flight behavior of a single UAV, while some other simulators can manage several UAVs simultaneously, but not in real time, thus losing accuracy regarding the mobility pattern of the UAV. In this work we address such problem by introducing Arducopter Simulator (ArduSim), a novel simulation platform that allows controlling in soft real-time the flight and communications of multiple UAVs, being the developed protocols directly portable to real devices. Moreover, ArduSim includes a realistic model for the WiFi communications link between UAVs, which was proposed based on real experiments. The chances that two UAVs get close to each other during their flights is increasing as more and more of them populate our skies, causing concerns regarding potential collisions. Therefore, this thesis also proposes the Mission Based Collision Avoidance Protocol (MBCAP), a novel UAV collision avoidance protocol applicable to all types of multicopters flying autonomously. It relies on wireless communications in order to detect nearby UAVs, and to negotiate the procedure to avoid any potential collision. Experimental and simulation results demonstrate the validity and effectiveness of the proposed solution, which typically introduces a small overhead in the range of 15 to 42 seconds for each risky situation successfully handled. The previous solution aims at UAVs performing independent flights, but they can also form a swarm, where more constraints have to be met to avoid collisions among them, and to allow them to complete their task efficiently. Deploying an UAV swarm instead of a single UAV can provide additional benefits when, for example, cargo carrying requirements exceed the lifting power of a single UAV, or when the deployment of several UAVs simultaneously can accelerate the accomplishment of the mission, and broaden the covered area. To this aim, in this work we present the Mission-based UAV Swarm Coordination Protocol (MUSCOP), a solution that allows multiple UAVs to perfectly coordinate their flight when performing planned missions. Experimental results show that the proposed protocol is able to achieve a high degree of swarm cohesion independently of the flight formation adopted, and even in the presence of very lossy channels, achieving minimal synchronization delays and very low position offsets with regard to the ideal case. Currently, there are some other scenarios, such as search and rescue operations, where the deployment of manually guided swarms of UAVs can be necessary. In such cases, the pilot's commands are unknown a priori (unpredictable), meaning that the UAVs must respond in near real-time to the movements of the leader UAV in order to maintain swarm consistency. Hence, in this thesis we also propose the FollowMe protocol for the coordination of UAVs in a swarm where the swarm leader is controlled by a real pilot, and the other UAVs must follow it in real-time to maintain swarm cohesion. Simulation results show the validity of the proposed swarm coordination protocol, detailing the responsiveness limits of our solution, and finding the minimum distances between UAVs to avoid collisions.[ES] A medida que la popularidad de los Vehículos Aéreos No Tripulados (VANTs) se incrementa, también se hacen necesarios nuevos protocolos para coordinarlos en vuelos sin control humano directo, y para evitar que colisionen entre sí. Probar estos nuevos protocolos en VANTs reales es un proceso complejo que requiere invertir mucho tiempo, dinero y esfuerzo investigador. Por lo tanto, es necesario probar en simulación las soluciones previamente implementadas. Lamentablemente, las herramientas actuales tienen importantes limitaciones: algunas simulan con precisión el vuelo de un único VANT, mientras que otros simuladores pueden gestionar varios VANTs simultáneamente aunque no en tiempo real, perdiendo por lo tanto precisión en el patrón de movilidad del VANT. En este trabajo abordamos este problema introduciendo Arducopter Simulator (ArduSim), una nueva plataforma de simulación que permite controlar en tiempo real el vuelo y la comunicación entre múltiples VANTs, permitiendo llevar los protocolos desarrollados a dispositivos reales con facilidad. Además, ArduSim incluye un modelo realista de un enlace de comunicaciones WiFi entre VANTs, el cual está basado en el resultado obtenido de experimentos con VANTs reales. La posibilidad de que dos VANTs se aproximen entre sí durante el vuelo se incrementa a medida que hay más aeronaves de este tipo surcando los cielos, introduciendo peligro por posibles colisiones. Por ello, esta tesis propone Mission Based Collision Avoidance Protocol (MBCAP), un nuevo protocolo de evitación de colisiones para VANTs aplicable a todo tipo de multicópteros mientras vuelan autónomamente. MBCAP utiliza comunicaciones inalámbricas para detectar VANTs cercanos y para negociar el proceso de evitación de la colisión. Los resultados de simulaciones y experimentos reales demuestran la validez y efectividad de la solución propuesta, que introduce un pequeño aumento del tiempo de vuelo de entre 15 y 42 segundos por cada situación de riesgo correctamente resuelta. La solución anterior está orientada a VANTs que realizan vuelos independientes, pero también pueden formar un enjambre, donde hay que cumplir más restricciones para evitar que colisionen entre sí, y para que completen la tarea de forma eficiente. Desplegar un enjambre de VANTs en vez de uno solo proporciona beneficios adicionales cuando, por ejemplo, la necesidad de carga excede la capacidad de elevación de un único VANT, o cuando al desplegar varios VANTs simultáneamente se acelera la misión y se cubre un área mayor. Con esta finalidad, en este trabajo presentamos el protocolo Mission-based UAV Swarm Coordination Protocol (MUSCOP), una solución que permite a varios VANTs coordinar perfectamente el vuelo mientras realizan misiones planificadas. Los resultados experimentales muestran que el protocolo propuesto permite al enjambre alcanzar un grado de cohesión elevado independientemente de la formación de vuelo adoptada, e incluso en presencia de un canal de comunicación con muchas pérdidas, consiguiendo retardos en la sincronización insignificantes y desfases mínimos en la posición con respecto al caso ideal. Actualmente hay otros escenarios, como las operaciones de búsqueda y rescate, donde el despliegue de enjambres de VANTs guiados manualmente puede ser necesario. En estos casos, las órdenes del piloto son desconocidas a priori (impredecibles), lo que significa que los VANTs deben responder prácticamente en tiempo real a los movimientos del VANT líder para mantener la consistencia del enjambre. Por ello, en esta tesis proponemos el protocolo FollowMe para la coordinación de VANTs en un enjambre donde el líder es controlado por un piloto, y el resto de VANTs lo siguen en tiempo real para mantener la cohesión del enjambre. Las simulaciones muestran la validez del protocolo de coordinación de enjambres propuesto, detallando los límites de la solución, y definiendo la distancia mínima entre VANTs para evita[CA] A mesura que la popularitat dels Vehicles Aeris No Tripulats (VANTs) s'incrementa, també es fan necessaris nous protocols per a coordinar-los en vols sense control humà directe, i per a evitar que col·lisionen entre si. Provar aquests nous protocols en VANTs reals és un procés complex que requereix invertir molt de temps, diners i esforç investigador. Per tant, és necessari provar en simulació les solucions prèviament implementades. Lamentablement, les eines actuals tenen importants limitacions: algunes simulen amb precisió el vol d'un únic VANT, mentre que altres simuladors poden gestionar diversos VANTs simultàniament encara que no en temps real, perdent per tant precisió en el patró de mobilitat del VANT. En aquest treball abordem aquest problema introduint Arducopter Simulator (ArduSim), una nova plataforma de simulació que permet controlar en temps real el vol i la comunicació entre múltiples VANTs, permetent portar els protocols desenvolupats a dispositius reals amb facilitat. A més, ArduSim inclou un model realista d'un enllaç de comunicacions WiFi entre VANTs, que està basat en el resultat obtingut d'experiments amb VANTs reals. La possibilitat que dues VANTs s'aproximen entre si durant el vol s'incrementa a mesura que hi ha més aeronaus d'aquest tipus solcant els cels, introduint perill per possibles col·lisions. Per això, aquesta tesi proposa Mission Based Collision Avoidance Protocol (MBCAP), un nou protocol d'evitació de col·lisions per a VANTs aplicable a tota mena de multicòpters mentre volen autònomament. MBCAP utilitza comunicacions sense fils per a detectar VANTs pròxims i per a negociar el procés d'evitació de la col·lisió. Els resultats de simulacions i experiments reals demostren la validesa i efectivitat de la solució proposada, que introdueix un xicotet augment del temps de vol de entre 15 i 42 segons per cada situació de risc correctament resolta. La solució anterior està orientada a VANTs que realitzen vols independents, però també poden formar un eixam, on cal complir més restriccions per a evitar que col·lisionen entre si, i perquè completen la tasca de forma eficient. Desplegar un eixam de VANTs en comptes d'un només proporciona beneficis addicionals quan, per exemple, la necessitat de càrrega excedeix la capacitat d'elevació d'un únic VANT, o quan en desplegar diversos VANTs simultàniament s'accelera la missió i es cobreix una àrea major. Amb aquesta finalitat, en aquest treball presentem el protocol Mission-based UAV Swarm Coordination Protocol (MUSCOP), una solució que permet a diversos VANTs coordinar perfectament el vol mentre realitzen missions planificades. Els resultats experimentals mostren que el protocol proposat permet a l'eixam aconseguir un grau de cohesió elevat independentment de la formació de vol adoptada, i fins i tot en presència d'un canal de comunicació amb moltes pèrdues, aconseguint retards en la sincronització insignificants i desfasaments mínims en la posició respecte al cas ideal. Actualment hi ha altres escenaris, com les operacions de cerca i rescat, on el desplegament d'eixams de VANTs guiats manualment pot ser necessari. En aquests casos, les ordres del pilot són desconegudes a priori (impredictibles), el que significa que els VANTs han de respondre pràcticament en temps real als moviments del VANT líder per a mantindre la consistència de l'eixam. Per això, en aquesta tesi proposem el protocol FollowMe per a la coordinació de VANTs en un eixam on el líder és controlat per un pilot, i la resta de VANTs ho segueixen en temps real per a mantindre la cohesió de l'eixam. Les simulacions mostren la validesa del protocol de coordinació d'eixams proposat, detallant els límits de la solució, i definint la distància mínima entre VANTs per a evitar col·lisions.Fabra Collado, FJ. (2020). Flight coordination solutions for multirotor unmanned aerial vehicles [Tesis doctoral no publicada]. Universitat Politècnica de València. https://doi.org/10.4995/Thesis/10251/147857TESI

Vision-based Marker-less Landing of a UAS on Moving Ground Vehicle

In recent years the use of unmanned air systems (UAS) has seen extreme growth. These small, often inexpensive platforms have been used to aid in tasks such as search and rescue, medicinal deliveries, disaster relief and more. In many use cases UAS work alongside unmanned ground vehicles (UGVs) to complete autonomous tasks. For end-to-end autonomous cooperation, the UAS needs to be able to autonomously take off and land on the UGV. Current autonomous landing solutions often use fiducial markers to aid in localizing the UGV relative to the UAS, an external ground computer to aid in computation, or gimbaled cameras on-board the UAS. This thesis seeks to demonstrate a vision-based autonomous landing system that does not rely on the use of fiducial markers, completes all computations on-board the UAS, and uses a fixed, non-gimbaled camera. Algorithms are tailored towards low size, weight, and power constraints as all compute and sensing components weigh less than 100 grams. The foundation of this thesis extends upon current efforts by localizing the UGV relative to the UAS using neural network object detection and camera intrinsic properties instead of common place fiducial markers. An object detection neural network is used to detect the UGV within an image captured by the camera on-board the UAS. Then a localization algorithm utilizes the UGV’s pixel position within the image to estimate the UGV’s position relative to the UAS. This estimated position of the UGV will be passed into a command generator that sends setpoints to the on-board PX4 flight control unit (FCU). This autonomous landing system was developed and validated within a high-fidelity simulation environment before conducting outdoor experiments

Impact of UAV Hardware Options on Bridge Inspection Mission Capabilities

Uncrewed Aerial Vehicles (UAV) constitute a rapidly evolving technology field that is becoming more accessible and capable of supplementing, expanding, and even replacing some traditionally manual bridge inspections. Given the classification of the bridge inspection types as initial, routine, in-depth, damage, special, and fracture critical members, specific UAV mission requirements can be developed, and their suitability for UAV application examined. Results of a review of 23 applications of UAVs in bridge inspections indicate that mission sensor and payload needs dictate the UAV configuration and size, resulting in quadcopter configurations being most suitable for visual camera inspections (43% of visual inspections use quadcopters), and hexa- and octocopter configurations being more suitable for higher payload hyperspectral, multispectral, and Light Detection and Ranging (LiDAR) inspections (13%). In addition, the number of motors and size of the aircraft are the primary drivers in the cost of the vehicle. 75% of vehicles rely on GPS for navigation, and none of them are capable of contact inspections. Factors that limit the use of UAVs in bridge inspections include the UAV endurance, the capability of navigation in GPS deprived environments, the stability in confined spaces in close proximity to structural elements, and the cost. Current research trends in UAV technologies address some of these limitations, such as obstacle detection and avoidance methods, autonomous flight path planning and optimization, and UAV hardware optimization for specific mission requirements

Implementation and Validation of a High Accuracy UAV-Photogrammetry Based Rail Track Inspection System

The regular inspection of the crane tracks of storage cranes at the Container Terminal Altenwerder (CTA), Hamburg requires high accuracy of measurements to determine its position. The allowed tolerances are in the range of 10 mm in the XY plane on a track length of 300 m. The traditional semi-automatic surveying methods are slow and require the interruption of the activities in the storage blocks. The research project AeroInspekt proposed a fully automatic measurement of the position of the tracks using UAV-based photogrammetry. In this paper, the results of the test campaign, carried out in June 2020, were presented where different cameras (150 mm and 80 mm telelens) and flight speeds (1.1 m/s and 1.9 m/s) at a 35 m flying height were performed. Furthermore, an automated rail delineation in the derived surface model was developed and evaluated with ground reference measurements. The results show that the required accuracy of the rail position with an RMSE of 3 mm in XY plane and 8 mm in altitude can be achieved with comparatively less disruption of regular block activities