The use of autonomous landing of aerial vehicles is increasing in demand. Applications of this ability can range from simple drone delivery to unmanned military missions. To be able to land at a spot identified by local information, such as a visual marker, creates an efficient and versatile solution. This allows for a more user/consumer friendly device overall. To achieve this goal the use of computer vision and an array of ranging sensors will be explored. In our approach we utilized an April Tag as our location identifier and point of reference. MATLAB/Simulink interface was used to develop the platform environment

Bitencourt, Gabriel

Brown, Elijah J

English

Sacred Heart University: DigitalCommons@SHU

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE Autonomous Aerial Vehicle Vision and Sensor Guided Landing Gabriel Bitencourt  School of Computer Science and Engineering                              Jack Welch College of Business  Sacred Heart University Fairfield, United States           Bitencourtg5@mail.sacredheart.edu                                                   Elijah Brown  School of Computer Science and Engineering                              Jack Welch College of Business  Sacred Heart University Fairfield, United States           Browne4@mail.sacredheart.edu                                                     Cedric Bleimling School of Computer Science and Engineering                              Jack Welch College of Business  Sacred Heart University Fairfield, United States           bleimlingc@sacredheart.edu                                                                    Gilbert Lai, PhD Research and Development Engineer                          Research Software Quanser Markham, Canada Gilbert.Lai@Quanser.com    Arman Molki Application Marketing Engineer                        Academic Applications    Quanser Markham, Canada Arman.Molki@Quanser.com Tolga Kaya, PhD School of Computer Science and Engineering                              Jack Welch College of Business  Sacred Heart University Fairfield, United States           kayat@sacredheart.edu      Abstract— The use of autonomous landing of aerial vehicles is increasing in demand. Applications of this ability can range from simple drone delivery to unmanned military missions. To be able to land at a spot identified by local information, such as a visual marker, creates an efficient and versatile solution. This allows for a more user/consumer friendly device overall. To achieve this goal the use of computer vision and an array of ranging sensors will be explored. In our approach we utilized an April Tag as our location identifier and point of reference. MATLAB/Simulink interface was used to develop the platform environment. I. INTRODUCTION (HEADING 1)         In recent years, the world has witnessed a rapid development of Unmanned Aerial Vehicles (UAV’s), and their potential in becoming the primary source of delivery in many places. Companies such as Amazon, Walmart, and UPS are working toward being among the first companies to commercially fly drones full time [1]-[3]. Given the fact that society is in the midst of a global pandemic, now seems like the perfect time for one of these companies to take that extra leap. In a time where citizens, specifically those over the age of 60, are being urged to stay inside, it can become difficult for elderly people to retrieve their daily necessities (groceries, medication, etc.) [4]. Aside from the pandemic, elderly already face difficulty obtaining their daily essentials. Now when accompanied by the fact that simple human interaction puts them at an increased risk of death due to COVID-19, the potential impact of drone delivery becomes immediately apparent.         Recently, Amazon and UPS have been granted Part 135 air carrier certificates by the Federal Aviation Administration which will give them the clearance to further troubleshoot and develop their autonomous drone fleets [1]-[2]. Walmart has been making progress of its own since they partnered with Flytrex, an autonomous drone delivery startup, and the two commenced their collaboration by delivering select grocery and household items from Walmart locations in Fayetteville, North Carolina [3]. Walmart has since upgraded its experimentation to delivering COVID-19 tests in the area around its store location in North Las Vegas [4]-[5].   Companies such as Quanser and DJI have also developed drone technology on their own, but for students and researchers seeking to learn the mechanics behind them. We Quanser and DJI offer to educational institutions, and solely utilized the environment developed by Quanser for this particular project. Quanser offers state-of-the-art quadrotor UAV’s and a multi-PC control station, and the drones communicate through Simulink ®  and MATLAB ®  coding, which many engineering students use daily. In this work, we utilized image processing and a ranging system for localization purposes. For image processing we used an April Tag as location marker. An April Tag is square with a combination of different black and white squares within it. These smaller squares act as data points at which a computer can access different information based on the image comprised of different patterns with in a 8 x 8 matrix like arrangement [6]-[8]. The ranging system would be used in calculating the height. With the use of OptiTrack ®  Motion Capture System we are given the bounds of operation which in turn allows the drone to know its relative location to the designated landing zone as well as the information necessary in order for the drone to move in respect to a 2-D Cartesian plane.   The drone searches for a location marker known as an Apriltag following a increasing square pattern. The drone begins at the center of the workspace. If it has done this without detecting the tag, it will slightly increase its distance from the center and trace the new size. It will continue to do this until the tag is detected or until its edges of the workspace is reached. We have created a Simulink ®  model to output: whether a tag was found, how many tags were found, a matrix containing the location of the squares, and the 4 corners of the tag. With this Information and knowledge of the original size of the image, the drone can know its relative location and orientation of its landing area. The ranging system would be used in the deduction of height calculation, and the OptiTrack ®  Motion Capture System will determine the surrounding proximity for safety. II. BACKGROUND A. Quanser Autonomous Vehicle The Quanser Autonomous Vehicle Research Studio (AVRS) consists of 8 motion capture cameras covering a 20 x 20 foot area. The drones themselves are equipped with reflective tracking spheres. Once calibrated, the cameras create a 3D field of operation. The cameras simulate a GPS allowing for easier steps towards real world implementation. The ground station consists of a three-monitor computer system, and it is from here that the Quanser Studio is controlled. Quanser has developed its own software, called QUARC, that enables users to create, build and deploy commands and applications created in Simulink onto the target hardware.  B. QDrone The Quanser Qdrone comes with multiple 640x480 resolution cameras, however, only the camera mounted on the underside of the drone is utilized. This, combined with the strong Intel® Aero Compute Board housed inside the drone, enables our ability to obtain data on the AprilTag location. An algorithm is used to feed the drone waypoints based on data received from the image processing, and it is done so in an array of indices with inputs as [x y z r]. The inputs “x”, “y” and “z” are read in meters and “r” denotes radians, pertaining to the direction in which the drone points. However, the algorithm’s command that tells the drone where to fly is unknown as a waypoint.  C. April Tag The April Tag was developed as a location marker with a barcode like appearance from April Robotics Laboratory at University of Michigan [7]. The concept behind their algorithm for tag detection is based around line-segment detection through analyzing pixel location. The direction of pixel clusters is graphed with the goal of determining length and magnitude of lines which in turn are grouped in order to output the tag ID recognized and relative location through size and angle of the image captured [7]. Through the MATLAB/Simulink toolbox known as QUARC a Tag Detection optimized for the use of April Tag detection computes and outputs a variety of information including but not limited to if a tag is found, number of tags found, the ID number of the tag(s), the pixel location of the center of tag found, and pixel location of the 4 corners of the tag found [6].  III. APRIL TAG LOCATION ALGORITHM After the camera image is processed, the whereabouts of the drone relative to tag is determined by a combination of the pixel location for the center of tag found and taking the place of GPS, the feedback of the workspace cameras as to the current location of the drone.  A. Calibration It is necessary to determine the physical size mapping of a pixel from the onboard camera in terms of meters in which the measurement system of the first 3 inputs of a waypoint [x y z]. Our calibration method is based on the drone being held at a constant 1-meter height (z = 1 m) and using the pixel coordinates outputted by the Tag Detection block. The calibration is performed in two parts known as finding PC and PR, which represents the size of a pixel in terms of meters. In order for consistency while calculating the pixel to meter ratio, we built a solid wooden rig to hold the drone at the desired 1-meter height simulating the drone in a hovering state. The center of the drone’s onboard camera was marked on the baseplate for the rig. The tag was placed at a measured distance denoted as DA from the center mark. CAT is the pixel location for the center of the April Tag. CDC is the pixel location of the drone camera’s center. The pixel size at 1-meter height is denoted as PC and is equal to DA divided by the difference of CAT and CDC shown in equation (1). TLT and BRT are the pixel locations given for top left corner and bottom right corner of the April Tag respectively . The distance measured between top left corner and bottom right corner of the April Tag with a ruler is denoted as DCM shown in equation (2). PR is equal to DCM divided by the difference of TLT and BRT. An average between PC and PR is then taken giving the value of PS which is pixel size in meters used in the main algorithm shown in equation (3). In our test the value for PC equaled 0.0023 m and PR equaled 0.0031 m. Thus, PS which is the average of PC and PR equaled 0.0027 m. Fig. 1      Above displays an example of an April Tag family 36h11 tag 0    PC = DA / (CAT – CDC)  (1)  PR = DCM / (TLT – BRT) (2)  PS = (PC + PR) / 2 (3)  B. Tag Algorithm When the tag algorithm is mentioned it is in reference to a MATLAB function block where the bulk of calculations is computed. The function takes in the inputs of the row and column pixel coordinate denoted as CR and CC given by the Tag Detection block communicated from the drone. The absolute value of the difference between CC and 240 is then multiplied by PS in order to get the X coordinate shown in equation (4).  The absolute value of the difference between CR and 320 is then scaled by PS in order to get the Y coordinate shown in equation (5).    X = |  (CC -  240 ) * PS  | (4)  Y = |  (CR -  320 ) * PS  |  (5)    Due to the constraint of the camera on the drone itself being mounted in a horizontal direction, a section of the algorithm using If Else statements are used to determine the true quadrant location of the tag relative to the drone. Through those statements the X coordinate and Y coordinate are then multiple by the appropriate 1 and -1 version. Example: if tag is detected in quadrant 2, X is multiplied by 1 and Y is multiplied by -1. The output known as Goto which is then added with the current position given by the workspace cameras determines the waypoint/ coordinates of where the drone will travel to in order to hover over the tag and ultimately land.  Fig. 3. Onboard camera view of quadrant and drone flight direction displayed with an AprilTag in sight Implimentation and results IV. IMPLIMENATION AND RESULTS A. Implimentation The drone will begin its scan with a typical searching sequence. It will either be a square pattern that increments continuously, starting small initially and gradually becoming bigger with time. The drone has been programmed with the help of Simulink coding and QUARC software to detect and land on its target, an April Tag. The drone will follow its predetermined path until its camera indicates that the tag has been found. At this instance, a switch will be triggered, and the drone will begin following an algorithm that computes the remaining distance between the drone and the tag so that the drone can land in the appropriate place.   B. Intial Results In testing, with calibration value PS at 1 pixel equals 0.0023 meters starting from the center of the workspace without the spiral pattern initiated, the drone successfully flies to the direction of the tag. The spiral algorithm is to be implemented with the tag then placed in a random location away from the drone. Part of our tests include giving the drone a default location of (0,0) in place of a spiral pattern to follow is the Boolean flag of whether a tag is detected or not is in false state. The tag was placed at the top of quadrant 1 from the camera's view and as the drone flew past the tag and lost sight it returned to (0,0). The reason for the overshooting of the drone is still being determined.    Y X -X -Y Front  Fig. 2      Above displays the points of interest in reference to the equations (1)-(3). The top shows the drone point of view and its center relative to the center of the April Tag. In the bottom it shows the drone on the test rig and its distance from the April Tag. 1 m  Fig. 4. Displays the moment in which sight of the tag is lost. V. FUTURE WORK  The use of the ranging sensors will become vital in the next stages. The ranging sensors will give the drones height in turn replacing the need of the constant of 1 m used in calibration intiallally. The sensors would also aid in taking into account the size of the package so when the drone goes to land, the package is secure and not crushed.   REFERENCES  [1] C. de Leon, “Drone Delivery? Amazon Moves Closer With F.A.A. Approval”, The New York Times, August 31, 2020. [Online]. Available: https://www.nytimes.com/2020/08/31/business/amazon-drone-delivery.html [Accessed October 5, 2020]. [2] “UPS Flight Forward Attains FAA’s First Full Approval For Drone Airline”, UPS Pressroom, October 1, 2019. [Online]. Available: https://pressroom.ups.com/pressroom/ContentDetailsViewer.page?ConceptType=PressReleases&id=1569933965476-404 [Accessed October 5, 2020]   [3]  T. Ward, “Walmart Now Piloting On-Demand Drone Delivery with Flytrex”, Walmart, September 9, 2020. [Online]. Available: https://corporate.walmart.com/newsroom/2020/09/09/walmart-now-piloting-on-demand-drone-delivery-with-flytrex [Accessed October 5, 2020]. [4] L. Maragakis, “Coronavirus and COVID-19: Who is at higher risk?,” Coronavirus and COVID-19: Who is at higher risk? | Johns Hopkins Medicine, 2020. [Online]. Available: https://www.hopkinsmedicine.org/health/conditions-and-diseases/coronavirus/coronavirus-and-covid19-who-is-at-higher-risk. [Accessed: 06-Oct-2020].   [5] M. Egan, “Walmart is using drones to deliver Covid-19 tests”, CNN, September 23, 2020. [Online]. Available: https://www.cnn.com/2020/09/23/business/covid-test-drone-delivery-walmart/index.html [Accessed October 5, 2020].   [6] Quanser-update.azurewebsites.net. 2020. Image Find Tags :: QUARC Targets Library. [online] Available at: <http://quanser-update.azurewebsites.net/quarc/documentation/image_find_tags_block.html> [Accessed 6 October 2020]. [7] E. Olson, AprilTag: A robust and flexible visual fiducial system, 2011. [Online]. Available: https://april.eecs.umich.edu/media/pdfs/olson2011tags.pdf. [Accessed: 2020].  [8] J. Wang and E. Olson, AprilTag 2: Efficient and robust fiducial detection, 2016. [Online]. Available: https://april.eecs.umich.edu/media/media/pdfs/wang2016iros.pdf. [Accessed: 2020].  [9] A. Arbeit, Adaptation and Automation of Search and Rescue Patterns with Implementation for an Operational Unmanned Aircraft System, 2013. [Online]. Available: https://digital.lib.washington.edu/researchworks/bitstream/handle/1773/25094/Arbeit_washington_0250O_12399.pdf?isAllowed=y&sequence=1. [Accessed: 2020].   . 

Autonomous Aerial Vehicle Vision and Sensor Guided Landing

https://digitalcommons.sacredheart.edu/cgi/viewcontent.cgi?filename=1&article=1617&context=acadfest&type=additional

Autonomous Aerial Vehicle Vision and Sensor Guided Landing

Abstract

Similar works

Full text

Available Versions

Sacred Heart University: DigitalCommons@SHU