Recent advances in robotics and nanotechnology resulted in a set of miniaturized Unmanned Aerial Vehicles (UAVs). Such small UAVs are envisioned to operate in hard-to-reach areas for enabling applications such as structural monitoring or content capturing. Towards showcasing this vision, we demonstrate a small UAV-supported setup for real-time autonomous generation of 3-Dimensional (3D) representations of static objects. In the setup, a small UAV (i.e., CrazyFlie 2.1) is envisioned to visit a set of locations, acting as a carrier and power source of a camera sensor. At each location, the sensor is expected to take a picture of the object and report it to the station. The station implements a pipeline for 3D reconstruction based on the pictures taken by the UAV.This work was funded by the MCIN/AEI/10.13039/01100011033/ FEDER/EU HoloMit 2.0 (PID2021-126551OB-C21) and MCIN/AEI/ 10.13039/501100011033 TRAINER-A (PID2020-118011GB-C21).Peer ReviewedPostprint (author's final draft

Abadal Cavallé, Sergi

Costa Pérez, Xavier

Lemic, Filip

Ollé Ferrayuoli, Bernat

Talarn Bell-lloch, Pau

English

UPCommons. Portal del coneixement obert de la UPC

Real-time Generation of 3-Dimensional Representations of StaticObjects using Small Unmanned Aerial VehiclesPau Talarn∗NaNoNetworking Center inCatalunya (N3Cat), UniversitatPolitècnica de Catalunya, Spainpau.talarn@estudiantat.upc.eduBernat Ollé∗NaNoNetworking Center inCatalunya (N3Cat), UniversitatPolitècnica de Catalunya, Spainbernat.olle@estudiantat.upc.eduFilip Lemic†AI-driven Systems Lab, i2CatFoundation, Spainfilip.lemic@i2cat.netSergi AbadalNaNoNetworking Center inCatalunya (N3Cat), UniversitatPolitècnica de Catalunya, Spainsergi.abadal@upc.eduXavier Costa PerezAI-driven Systems Lab, i2CatFoundation, Spain; NEC LaboratoriesEurope, Germany; ICREA, Spainxavier.costa@i2cat.netABSTRACTRecent advances in robotics and nanotechnology resulted in a set ofminiaturized Unmanned Aerial Vehicles (UAVs). Such small UAVsare envisioned to operate in hard-to-reach areas for enabling appli-cations such as structural monitoring or content capturing. Towardsshowcasing this vision, we demonstrate a small UAV-supportedsetup for real-time autonomous generation of 3-Dimensional (3D)representations of static objects. In the setup, a small UAV (i.e.,CrazyFlie 2.1) is envisioned to visit a set of locations, acting as acarrier and power source of a camera sensor. At each location, thesensor is expected to take a picture of the object and report it to thestation. The station implements a pipeline for 3D reconstruction ofthe object based on its pictures taken by the UAV.ACM Reference Format:Pau Talarn, Bernat Ollé, Filip Lemic, Sergi Abadal, and Xavier Costa Perez.2023. Real-time Generation of 3-Dimensional Representations of StaticObjects using Small Unmanned Aerial Vehicles. In Proceedings of ACMConference (Conference’17). ACM, New York, NY, USA, 3 pages. https://doi.org/10.1145/nnnnnnn.nnnnnnn1 INTRODUCTIONRecent advances in robotics resulted in a rapid uptake of the Un-manned Aerial Vehicle (UAV) technology [4]. At the same time,recent advances in nanotechnology are driving the miniaturizationof advanced systems such as UAVs [1]. This resulted in miniaturizedsystems known as small UAVs, such as the CrazyFlie 2.1 depictedin Figure 1. Small UAVs are opening the space for a novel line of∗The authors contributed equally to this work.†Corresponding author.Permission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citationon the first page. Copyrights for components of this work owned by others than ACMmust be honored. Abstracting with credit is permitted. To copy otherwise, or republish,to post on servers or to redistribute to lists, requires prior specific permission and/or afee. Request permissions from permissions@acm.org.Conference’17, July 2017, Washington, DC, USA© 2023 Association for Computing Machinery.ACM ISBN 978-x-xxxx-xxxx-x/YY/MM. . . $15.00https://doi.org/10.1145/nnnnnnn.nnnnnnnapplications targeting hard-to-reach areas for automatizing struc-tural monitoring, generating Virtual Reality (VR) content, etc. Anadditional benefit of small UAV systems comes from the fact thatthey are mainly envisioned to be utilized indoors and, as such, theywill mostly not require flight licenses. They are, therefore, more in-teresting from the average user perspective than the outdoor UAVsand, hence, feature a higher scaling potential and market uptake.One application that has relatively recently generated a signif-icant interest is the digital reconstruction of 3-Dimensional (3D)objects [7], finding its utility advertisement, gaming, hospitality, etc.Although we are witnessing a surge of such applications on devicessuch as smartphones, they still require manual interventions, e.g.,taking pictures from different angles, to be able to generate 3Dobject representations. As such, these approaches consume timeand effort, consequently not scaling well.In contrast, in this demonstration we show that such applica-tions can be enabled through the utilization of small-scale UAVs. Inparticular, we demonstrate an open-source pipeline consisting ofa small UAV acting as a carrier and power source of a Red, Greenand Blue (RGB) camera sensor. The UAV can sequentially visit aset of locations around the object of interest and hover at eachof them for taking a picture of the object. This is followed by thepicture being transferred wirelessly to a station that features anopen-source pipeline for generating a 3D representation of thetarget object. The sequence of locations to be visited by the UAVis determined dynamically based on the current features of thegenerated representation (i.e., a pointcloud) of the object.2 SYSTEM DESIGN AND IMPLEMENTATIONSystem Design. The setup for the generation of 3D representa-tions of static objects is depicted in Figure 2. We envision the UAVto be positioned for take-off facing the object. The generation of3D reconstruction is then triggered from the station, which resultsin the UAV taking off and capturing the first pictures. We use anRGB camera sensor soldiered on the UAV for capturing the pictures.The utilized camera sensor system also features a WiFi transceiver,which is used for communication with the station (i.e., requestingthe system to take a picture and transmitting picture to the station).The final publication is available at ACM via http://dx.doi.org/10.1145/3570361.3614078Conference’17, July 2017, Washington, DC, USA P. Talarn, B. Ollé, F. Lemic, S. Abadal, X. Costa PerezFigure 1: CrazyFlie 2.1Figure 2: Setup for small UAV-supported generation of 3D object representationsTwo images taken from two different locations are needed forproducing the initial pointcloud representation of the object, hencethe UAV is initially instructed to take two images of the objectdirectly in front of it, with a slight drift in the locations. The point-cloud is then generated using the well-known Structure from Mo-tion (SfM) approach [6]. The pointcloud generation is preceded bythe background removal from the pictures, as it is well known thatthe background features negatively affect the pointcloud generationaccuracy. The SfM algorithm operates on the principle of obtainingthe location and orientation of a monocular camera with respect toa target object, based on which it is able to reconstruct the object.Based on the available pointcloud, we utilize the Kernel DensityEstimation (KDE) algorithm for deciding on the next location to bevisited by the UAV. The KDE algorithm uses a mixture consisting ofone Gaussian component per point, resulting in an essentially non-parametric estimator of density. The intuition behind this approachis to minimize the time needed for obtaining satisfactory pointcloudquality by positioning the UAV in a way that covers the pointcloudregions with lowest density.System Implementation. Weutilize Crazyflie 2.1 UAV in our demon-stration, primarily as it is an open hardware and software platform.Crazyflie 2.1 UAVs come with a FreeRTOS operating system anda Bluetooth LE transceivers for control. They also feature an ac-celerometer, gyroscope, magnetometer, high precision pressuresensor, and two expansion boards for capability extensions. In thiswork, one expansion slot is utilized for the Loco Positioning Sys-tem (LPS) tag, while the RGB camera sensor (i.e., an ESP32-CAMsystem) is connected solely to the powering pins of the UAV.We use the Crazyflie’s LPS to provide accurate indoor position-ing. The system consist of a tag to be localized and multiple anchorsdistributed over the volume used for localizing the tag. The systemcan be deployed by simply positioning of the anchors, measuringtheir coordinates relative to a chosen origin, and initializing theirautomated calibration for synchronizing their transmission sched-ules. Once the anchors are self-calibrated, they can be used forlocalizing the UAV and consequently for its navigation toward theinstructed locations. Given that the system for 3D reconstruction isrelying solely on physical deployment, localization, and initiationof the self-calibration of the anchors, we argue that it can be rapidlydeployed in new environments.The LPS tag Decawave 1000 uses Ultra Wide-Band (UWB) tech-nology for communication and localization. The tag can estimateits location based on the UWB signals received from the anchors,where the localization is performed using the Time Difference ofArrival (TDoA) procedure. A Crazyflie equipped with an LPS deckmakes use of an extended Kalman filter to estimate its orientationand position, with its implementation based on [5]. The utilizedsystem with 8 anchors is able to localize the tag in a 3D space withcm-level accuracy at the range of about 10 m [2, 3].Software control of the UAV is done through an application thatmakes use of the Crazyflie Python library. It runs the followingsequence: i) initialization and UAV take-off, ii) taking two pictureswith a small offset in location, iii) sending pictures for pointcloudgeneration, iv) move to the next waypoint (defined as an <x, y, z>tuple) based on the quality of the currently generated pointcloud,v) taking and transmitting a new picture, etc. Communicating withand controlling the Crazyflie remotely can be done using a customUSB dongle called the Crazyradio, which provides 126 channels(2400 MHz - 2525 MHz) and works with the custom CrazyradioRealTime Protocol (CRTP) to communicate with the UAV.3 CONCLUSIONIn this demonstration1,2, we will showcase a system for small Un-manned Aerial Vehicle (UAV)-supported autonomous real-time gen-eration of 3-Dimensional (3D) representations (i.e., pointclouds) ofstatic objects. Our aim is to receive an early community feedbackon the system, given that we envision it as a driver for our futureresearch. In particular, we envision eventually switching from acamera sensor to high-frequency antenna array that can providecomparable pointcloud generation performance, yet with by-designbenefits along the lines of reduced energy consumption, privacypreservation, and reduced communication latency. Moreover, fea-tures such as dynamic obstacle avoidance and in-fleet operation areenvisioned to be introduced. In terms of the vision, we argue thatsuch an enhanced system will eventually be able to autonomouslygenerate accurate real-time 3D representations of moving objectsand people, providing a primer for a novel set of applications (e.g.,person capturing system for fully-immersive Virtual Reality (VR)).1Demonstration video: to be provided at the camera-ready stage.2Code repository: https://bitbucket.org/bernat-i2cat/3d_reconstruction/src/master/Real-time Generation of 3D Representations of Static Objects using Small UAVs Conference’17, July 2017, Washington, DC, USAREFERENCES[1] Filip Lemic et al. 2021. Survey on terahertz nanocommunication and networking:A top-down perspective. IEEE Journal on Selected Areas in Communications 39, 6(2021), 1506–1543.[2] Ken Mendes et al. 2020. Automated, autonomous, and repeatable wireless ex-perimentation in heterogeneous 3D environments: demo abstract. In EmbeddedNetworked Sensor Systems (SenSys). ACM, 601–602.[3] Ken Mendes et al. 2022. Small UAVs-supported Autonomous Generation of Fine-grained 3D Indoor Radio Environmental Maps. In Distributed Computing SystemsWorkshops (ICDCSW). IEEE, 296–301.[4] Mohammad Mozaffari et al. 2021. Toward 6G with connected sky: UAVs andbeyond. IEEE Communications Magazine 59, 12 (2021), 74–80.[5] Mark W Mueller et al. 2015. Fusing ultra-wideband range measurements withaccelerometers and rate gyroscopes for quadrocopter state estimation. In Roboticsand Automation (ICRA). IEEE, 1730–1736.[6] Onur Özyeşil, Vladislav Voroninski, Ronen Basri, and Amit Singer. 2017. A surveyof structure from motion. Acta Numerica 26 (2017), 305–364.[7] Elisavet Stathopoulou et al. 2019. Open-source image-based 3D reconstructionpipelines: Review, comparison and evaluation. Photogrammetry, Remote Sensingand Spatial Information Sciences, Volume XLII-2/W17 (2019), 331–338.

Real-time generation of 3-dimensional representations of static objects using small unmanned aerial vehicles

https://upcommons.upc.edu/bitstream/2117/397390/1/small_uavs_demo.pdf

Real-time generation of 3-dimensional representations of static objects using small unmanned aerial vehicles

Abstract

Similar works

Full text

Available Versions

UPCommons. Portal del coneixement obert de la UPC