A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones

Fully-autonomous miniaturized robots (e.g., drones), with artificial intelligence (AI) based visual navigation capabilities are extremely challenging drivers of Internet-of-Things edge intelligence capabilities. Visual navigation based on AI approaches, such as deep neural networks (DNNs) are becoming pervasive for standard-size drones, but are considered out of reach for nanodrones with size of a few cm${}^\mathrm{2}$. In this work, we present the first (to the best of our knowledge) demonstration of a navigation engine for autonomous nano-drones capable of closed-loop end-to-end DNN-based visual navigation. To achieve this goal we developed a complete methodology for parallel execution of complex DNNs directly on-bard of resource-constrained milliwatt-scale nodes. Our system is based on GAP8, a novel parallel ultra-low-power computing platform, and a 27 g commercial, open-source CrazyFlie 2.0 nano-quadrotor. As part of our general methodology we discuss the software mapping techniques that enable the state-of-the-art deep convolutional neural network presented in [1] to be fully executed on-board within a strict 6 fps real-time constraint with no compromise in terms of flight results, while all processing is done with only 64 mW on average. Our navigation engine is flexible and can be used to span a wide performance range: at its peak performance corner it achieves 18 fps while still consuming on average just 3.5% of the power envelope of the deployed nano-aircraft.Comment: 15 pages, 13 figures, 5 tables, 2 listings, accepted for publication in the IEEE Internet of Things Journal (IEEE IOTJ

GazeDrone: Mobile Eye-Based Interaction in Public Space Without Augmenting the User

Gaze interaction holds a lot of promise for seamless human-computer interaction. At the same time, current wearable mobile eye trackers require user augmentation that negatively impacts natural user behavior while remote trackers require users to position themselves within a confined tracking range. We present GazeDrone, the first system that combines a camera-equipped aerial drone with a computational method to detect sidelong glances for spontaneous (calibration-free) gaze-based interaction with surrounding pervasive systems (e.g., public displays). GazeDrone does not require augmenting each user with on-body sensors and allows interaction from arbitrary positions, even while moving. We demonstrate that drone-supported gaze interaction is feasible and accurate for certain movement types. It is well-perceived by users, in particular while interacting from a fixed position as well as while moving orthogonally or diagonally to a display. We present design implications and discuss opportunities and challenges for drone-supported gaze interaction in public

Beauty and the Beast: Optimal Methods Meet Learning for Drone Racing

Autonomous micro aerial vehicles still struggle with fast and agile maneuvers, dynamic environments, imperfect sensing, and state estimation drift. Autonomous drone racing brings these challenges to the fore. Human pilots can fly a previously unseen track after a handful of practice runs. In contrast, state-of-the-art autonomous navigation algorithms require either a precise metric map of the environment or a large amount of training data collected in the track of interest. To bridge this gap, we propose an approach that can fly a new track in a previously unseen environment without a precise map or expensive data collection. Our approach represents the global track layout with coarse gate locations, which can be easily estimated from a single demonstration flight. At test time, a convolutional network predicts the poses of the closest gates along with their uncertainty. These predictions are incorporated by an extended Kalman filter to maintain optimal maximum-a-posteriori estimates of gate locations. This allows the framework to cope with misleading high-variance estimates that could stem from poor observability or lack of visible gates. Given the estimated gate poses, we use model predictive control to quickly and accurately navigate through the track. We conduct extensive experiments in the physical world, demonstrating agile and robust flight through complex and diverse previously-unseen race tracks. The presented approach was used to win the IROS 2018 Autonomous Drone Race Competition, outracing the second-placing team by a factor of two.Comment: 6 pages (+1 references

Deep Drone Racing: From Simulation to Reality with Domain Randomization

Dynamically changing environments, unreliable state estimation, and operation under severe resource constraints are fundamental challenges that limit the deployment of small autonomous drones. We address these challenges in the context of autonomous, vision-based drone racing in dynamic environments. A racing drone must traverse a track with possibly moving gates at high speed. We enable this functionality by combining the performance of a state-of-the-art planning and control system with the perceptual awareness of a convolutional neural network (CNN). The resulting modular system is both platform- and domain-independent: it is trained in simulation and deployed on a physical quadrotor without any fine-tuning. The abundance of simulated data, generated via domain randomization, makes our system robust to changes of illumination and gate appearance. To the best of our knowledge, our approach is the first to demonstrate zero-shot sim-to-real transfer on the task of agile drone flight. We extensively test the precision and robustness of our system, both in simulation and on a physical platform, and show significant improvements over the state of the art.Comment: Accepted as a Regular Paper to the IEEE Transactions on Robotics Journal. arXiv admin note: substantial text overlap with arXiv:1806.0854

Dynamic simulation of forest management normative scenarios : the case of timber plantations in the southern Chile

International audienceAccounting for spatial issues (spatially explicit simulation, geographical amenities and advantages of land use 10 and cover changes, etc.) to build prospective scenarios is a crucial issue for better assessment of possible impacts on the environment. Such spatialized scenarios and their implications allow societies to reduce the uncertainty of the future by exploring various strategies for land use changes. Despite the wide diversity in existing scenario-building techniques, two different approaches can be distinguished (exploratory vs. normative) for their methodological implications. The originality in this study comes from the use of a relevant exploratory 15 (dynamic) approach to map normative scenarios which, in most cases, are represented throughout the combination of narratives and synchronic land use and cover maps. The objective of the article is to apply this dynamic exploratory simulation approach to spatialize normative scenarios within the framework of forest management in southern Chile. In the results, two contrasting images of the future are compared, with the preservation of native forests on one hand and the spread of exotic timber plantations on the other

Dronen hyödyntäminen Vilppulan vankilan alueella

Kauko-ohjattavien miehittämättömien ilma-alusten lennättäminen ei nykyään ole enää pelkästään harrastustoimintaa, vaan vakavasti otettava ja nopeasti kasvava toimiala. Kauko-ohjattavasta miehittämättömästä ilma-aluksesta kamera- ja ohjausjärjestelmineen eli RPAS-laitteesta (Remotely Piloted Aircraft System) käytetään tässä työssä myös yleisesti julkisuudessa käytettyä nimitystä drone. Tämä opinnäytetyö tehtiin Rikosseuraamuslaitokselle ja erityisesti Vilppulan avovankilalle. Toimeksiantajan ja siten myös opinnäytetyön tavoitteena oli selvittää mihin muuhun drone-teknologiaa voisi hyödyntää Vilppulan avovankilan alueella kuin valvonnan ja turvallisuuden lisäämiseen. Työn tavoitteena oli selvittää dronen lennättämiseen ja sen avulla kuvaamiseen vaikuttava lainsäädäntö ja pohtia uuden lainsäädännön tuomia haasteita. Yhtenä tavoitteena oli selvittää ja kerätä tietoa, jonka pohjalta voidaan laatia lentotyössä vaadittava toimintakäsikirja. Opinnäytetyön tarkoituksena oli käynnistää pilottiprojekti, johon kuului ensimmäisen dronen hankkiminen vankilan käyttöön, dronen lennättämiseen vaadittavien dokumenttien selvittäminen ja dronen muiden käyttökohteiden kartoittaminen. Tutkimusmenetelmä oli lainopillinen perustuen voimassaolevaan lainsäädäntöön ja Trafin määräykseen. Dronella kuvaaminen Vilppulan avovankilan alueella on lentotyötä ja toiminnan aloittamisesta on ilmoitettava Trafille. Voimassa olevien määräysten mukaan lentotyötä harjoittavan on pidettävä lentopäiväkirjaa, tehtävä lennätyskohtainen riskikartoitus ja laadittava toimintakäsikirja. Tässä ns. RPAS-toimintakäsikirjassa on mm. kartoitettava lentotyön mahdolliset riskit, keinot niiden vähentämiseen sekä ohjeistus häiriötilanteiden varalle. Trafin mukaan Vilppulan alueen lennätyksistä on mahdollista kirjata toimintakäsikirjaan standardiskenaariot eri lennätyskohteista, jolloin lennätyskohtainen ilmoitus on tehtävä vain poikkeuksellisista lennätyksistä. Tulossa oleva lainsäädäntö ei sinänsä tuo erityisiä haasteita dronen käyttöön, mutta jo nyt vaadittavan toimintakäsikirjan laatimiseen on syytä kiinnittää erityistä huomiota ja kuvata lentotyökohteet mahdollisimman tarkasti. Dronen avulla on mahdollista tehostaa valvontaa ja turvallisuutta muutoin hankalasti valvottavilla Vilppulan avovankilaan kuuluvilla laajoilla metsä-, vesistö- ja peltoalueilla. Alueeseen kuuluva kaikille avoin puutarha voi hyödyntää dronea niin kasvutilanteen seurannassa kuin myös omassa markkinoinnissaan. Arvokiinteistöjen kunnossapidon kartoitukseen dronen käyttö apuvälineenä tuo merkittäviä säästöjä sekä kustannuksiin että ajankäyttöön.The use of remotely piloted unmanned aircraft is no longer just a hobby, but a serious and fast-growing business. In this thesis remotely piloted unmanned aircraft equipped with camera and control system (RPAS equipment) is referred to as a drone, which is also a commonly used term by the media. This thesis was commissioned by the Criminal Sanctions Agency and, in particular, Vilppula prison. The aim of the commissioner and hence of the thesis was to find out where else the drone technology could be used in Vilppula prison area but to increase control and security. The aim of the thesis was to find out the legislation that affects flying a drone and the changes that are brought by the upcoming legislation. The purpose of the thesis was to launch a pilot project, including the purchase of the first drone for the use of Vilppula prison, drafting the documents required to fly the drone and discovering other use for the drone. One objective was to collect information needed for producing the RPAS manual, which is required for the various aerial work functions. The research method was juridical based on the effective legislation as well as Trafi’s regulation OPS M1-32. Filming with the drone in the Vilppula prison area is aerial work and therefore Trafi requires an advance notification on the use of remotely piloted aircraft. According to the regulations in force, the operator must keep a flight log, make a flight-specific safety assessment and draw up an operations manual. This RPAS Operations Manual must describe the identification of an operationally typical hazard, risk assessment and mitigation measures. According to Trafi, it is possible to describe standard scenarios for various operations in the Vilppula area in the operations manual, in which case a flight-specific assessment is to be made only for exceptional flights. The upcoming legislation does not pose any particular challenges to the introduction of the drone. However, it is important to pay special attention to the preparation of the RPAS Operations Manual and describe the aerial work functions as precisely as possible. With the drone, it is possible to enhance the control and safety of the otherwise difficultly controlled areas, such as the large forest area, coastline and the fields. Vilppula prison also runs /upholds a garden & gift shop and the drone can be used for taking photos to track the growth of the crop and for promotional use in their marketing. In finding out the possible maintenance needs of the listed buildings, the use of the drone as an auxiliary instrument provides significant time and cost savings

Everybody’s Neighborhood Stadium: Memory and Baseball in Newark, NJ

It starts with a shout, “You have to be kidding me!” Then another, “What! No way!” Then you hear from the upper deck something that is R-rated. The decibel level increases, it is ascending rapidly, soon the noise blankets the whole stadium and in that moment it could not be clearer, in all the variations of words encircling the stadium there’s agreement, the umpire is legally blind. There is no universe where a man with sight would call him safe at second. Something happens when you are in a stadium. It’s as if the outside world is suspended and life’s concerns shift. Here, you live and die by the swing of a bat, the arm of a pitcher, the glove of a fielder, and the eyes, 20/20 or not, of the umpires. Here, you are known by your allegiance, to the sport on the field and the team who is playing. Stadiums are spaces like no other. I know of a stadium that offered an escape from the outside world, but it’s gone now. It has been replaced with large concrete and steel rectangles spanning over most of the block it occupies with signs that say Concrete Systems Inc., D’Artagnan Gourmet Food Distributor, New Generation Iron Inc., Leopard Framing, and W.B. Law & Son. It’s gone now and the only trace of it left on Wilson Avenue between Avenue K and Avenue L in Newark, NJ is a plaque that was placed there by the Newark Preservation and Landmarks Committee in 2007.1 Ruppert Stadium housed an alternate reality, a baseball centered universe. It was a palace, a cathedral, a dream, a sanctuary, a memory, and a history of its own. At its foundation, Ruppert Stadium was a baseball stadium that from 1936 to 1948 was home of the Newark Bears, an all white New York Yankee farm team and the Newark Eagles, a member of the Negro Leagues. In the era of Jim Crow, in a city that is often defined by racial unrest, Ruppert Stadium was shared space. It was a space that brought citizens from all ethnic groups to its doors, well, its turnstiles. It was one of the few spaces that residents of Newark could call their own; their own neighborhood stadium. If stadiums function as a space where one reality disappears and another is created, then Ruppert Stadium produced a space where one could transcend temporarily the racial inequality that existed in the city. If we see as stadiums as “monuments, places for community interaction, repositories of collective memory, loci of strong identities, sites for ritualized conflict, political battlefields, and nodes in the global system of sport,”2 how do the memories of Ruppert Stadium as a racialized space, in both personal and collective or social memory3 shape the history being told of the Newark of the twentieth century? And how, over thirty years later with the building of a new minor league baseball stadium, does the legacy of Ruppert Stadium help define the Newark of the twenty-first century? 1 The Newark Preservation and Landmarks Committee (NPLC), “Newark Eagles Monument Dedicated,” http://newarkpreservation.blogspot.com/2010/10/newark-eagles-monument-dedicated.html. 2 Gaffney, Christopher Thomas Temples of the Earthbound Gods (Austin: University of Texas Press, 2008), 4. 3 Hayden, Dolores The Power of Place: Urban Landscapes as Public History (Cambridge: MIT Press, 1997), 9