Accelerating Reinforcement Learning by Composing Solutions of Automatically Identified Subtasks

This paper discusses a system that accelerates reinforcement learning by using transfer from related tasks. Without such transfer, even if two tasks are very similar at some abstract level, an extensive re-learning effort is required. The system achieves much of its power by transferring parts of previously learned solutions rather than a single complete solution. The system exploits strong features in the multi-dimensional function produced by reinforcement learning in solving a particular task. These features are stable and easy to recognize early in the learning process. They generate a partitioning of the state space and thus the function. The partition is represented as a graph. This is used to index and compose functions stored in a case base to form a close approximation to the solution of the new task. Experiments demonstrate that function composition often produces more than an order of magnitude increase in learning rate compared to a basic reinforcement learning algorithm

Battery Management Systems for Firefighting Robots Using Simulation Modeling

The battery management systems for firefighting robots are intended to enable firefighting robots to increase operating time and to effectively extinguish a fire while managing the amount of water in a fire hose and cooperating sub-robots. To increase the operating time by managing the traction power of the firefighting robot, a novel automatic T-valve device and sub-robots were designed and added to fire hoses. The main goal of the battery management systems for firefighting robots is to lower the weight of the fire hose and to increase traction power by working with sub-robots. Whenever a firefighting robot wants to move to other spaces, the battery management systems will remove the water from fire hoses and draw the empty fire hoses by using sub-robots; thus, they are able to help the main firefighting robot to carry lighter hoses and to operate for a longer time. As a result, the battery management systems for firefighting robots enable the firefighting robot to successfully extinguish a fire for a longer time and to efficiently reach the desired destinations. The demonstration will be modeled by a computer simulation program, called AnyLogic®, which can model a fire and fire areas and apply the battery management systems to robots in each fire site

A 64mW DNN-based Visual Navigation Engine for Autonomous Nano-Drones

Fully-autonomous miniaturized robots (e.g., drones), with artificial intelligence (AI) based visual navigation capabilities are extremely challenging drivers of Internet-of-Things edge intelligence capabilities. Visual navigation based on AI approaches, such as deep neural networks (DNNs) are becoming pervasive for standard-size drones, but are considered out of reach for nanodrones with size of a few cm${}^\mathrm{2}$. In this work, we present the first (to the best of our knowledge) demonstration of a navigation engine for autonomous nano-drones capable of closed-loop end-to-end DNN-based visual navigation. To achieve this goal we developed a complete methodology for parallel execution of complex DNNs directly on-bard of resource-constrained milliwatt-scale nodes. Our system is based on GAP8, a novel parallel ultra-low-power computing platform, and a 27 g commercial, open-source CrazyFlie 2.0 nano-quadrotor. As part of our general methodology we discuss the software mapping techniques that enable the state-of-the-art deep convolutional neural network presented in [1] to be fully executed on-board within a strict 6 fps real-time constraint with no compromise in terms of flight results, while all processing is done with only 64 mW on average. Our navigation engine is flexible and can be used to span a wide performance range: at its peak performance corner it achieves 18 fps while still consuming on average just 3.5% of the power envelope of the deployed nano-aircraft.Comment: 15 pages, 13 figures, 5 tables, 2 listings, accepted for publication in the IEEE Internet of Things Journal (IEEE IOTJ

Static and Dynamic Path Planning Using Incremental Heuristic Search

Path planning is an important component in any highly automated vehicle system. In this report, the general problem of path planning is considered first in partially known static environments where only static obstacles are present but the layout of the environment is changing as the agent acquires new information. Attention is then given to the problem of path planning in dynamic environments where there are moving obstacles in addition to the static ones. Specifically, a 2D car-like agent traversing in a 2D environment was considered. It was found that the traditional configuration-time space approach is unsuitable for producing trajectories consistent with the dynamic constraints of a car. A novel scheme is then suggested where the state space is 4D consisting of position, speed and time but the search is done in the 3D space composed by position and speed. Simulation tests shows that the new scheme is capable of efficiently producing trajectories respecting the dynamic constraint of a car-like agent with a bound on their optimality.Comment: Internship Repor

Exploration autonome et efficiente de chantiers miniers souterrains inconnus avec un drone filaire

Abstract: Underground mining stopes are often mapped using a sensor located at the end of a pole that the operator introduces into the stope from a secure area. The sensor emits laser beams that provide the distance to a detected wall, thus creating a 3D map. This produces shadow zones and a low point density on the distant walls. To address these challenges, a research team from the Université de Sherbrooke is designing a tethered drone equipped with a rotating LiDAR for this mission, thus benefiting from several points of view. The wired transmission allows for unlimited flight time, shared computing, and real-time communication. For compatibility with the movement of the drone after tether entanglements, the excess length is integrated into an onboard spool, contributing to the drone payload. During manual piloting, the human factor causes problems in the perception and comprehension of a virtual 3D environment, as well as the execution of an optimal mission. This thesis focuses on autonomous navigation in two aspects: path planning and exploration. The system must compute a trajectory that maps the entire environment, minimizing the mission time and respecting the maximum onboard tether length. Path planning using a Rapidly-exploring Random Tree (RRT) quickly finds a feasible path, but the optimization is computationally expensive and the performance is variable and unpredictable. Exploration by the frontier method is representative of the space to be explored and the path can be optimized by solving a Traveling Salesman Problem (TSP) but existing techniques for a tethered drone only consider the 2D case and do not optimize the global path. To meet these challenges, this thesis presents two new algorithms. The first one, RRT-Rope, produces an equal or shorter path than existing algorithms in a significantly shorter computation time, up to 70% faster than the next best algorithm in a representative environment. A modified version of RRT-connect computes a feasible path, shortened with a deterministic technique that takes advantage of previously added intermediate nodes. The second algorithm, TAPE, is the first 3D cavity exploration method that focuses on minimizing mission time and unwound tether length. On average, the overall path is 4% longer than the method that solves the TSP, but the tether remains under the allowed length in 100% of the simulated cases, compared to 53% with the initial method. The approach uses a 2-level hierarchical architecture: global planning solves a TSP after frontier extraction, and local planning minimizes the path cost and tether length via a decision function. The integration of these two tools in the NetherDrone produces an intelligent system for autonomous exploration, with semi-autonomous features for operator interaction. This work opens the door to new navigation approaches in the field of inspection, mapping, and Search and Rescue missions.La cartographie des chantiers miniers souterrains est souvent réalisée à l’aide d’un capteur situé au bout d’une perche que l’opérateur introduit dans le chantier, depuis une zone sécurisée. Le capteur émet des faisceaux laser qui fournissent la distance à un mur détecté, créant ainsi une carte en 3D. Ceci produit des zones d’ombres et une faible densité de points sur les parois éloignées. Pour relever ces défis, une équipe de recherche de l’Université de Sherbrooke conçoit un drone filaire équipé d’un LiDAR rotatif pour cette mission, bénéficiant ainsi de plusieurs points de vue. La transmission filaire permet un temps de vol illimité, un partage de calcul et une communication en temps réel. Pour une compatibilité avec le mouvement du drone lors des coincements du fil, la longueur excédante est intégrée dans une bobine embarquée, qui contribue à la charge utile du drone. Lors d’un pilotage manuel, le facteur humain entraîne des problèmes de perception et compréhension d’un environnement 3D virtuel, et d’exécution d’une mission optimale. Cette thèse se concentre sur la navigation autonome sous deux aspects : la planification de trajectoire et l’exploration. Le système doit calculer une trajectoire qui cartographie l’environnement complet, en minimisant le temps de mission et en respectant la longueur maximale de fil embarquée. La planification de trajectoire à l’aide d’un Rapidly-exploring Random Tree (RRT) trouve rapidement un chemin réalisable, mais l’optimisation est coûteuse en calcul et la performance est variable et imprévisible. L’exploration par la méthode des frontières est représentative de l’espace à explorer et le chemin peut être optimisé en résolvant un Traveling Salesman Problem (TSP), mais les techniques existantes pour un drone filaire ne considèrent que le cas 2D et n’optimisent pas le chemin global. Pour relever ces défis, cette thèse présente deux nouveaux algorithmes. Le premier, RRT-Rope, produit un chemin égal ou plus court que les algorithmes existants en un temps de calcul jusqu’à 70% plus court que le deuxième meilleur algorithme dans un environnement représentatif. Une version modifiée de RRT-connect calcule un chemin réalisable, raccourci avec une technique déterministe qui tire profit des noeuds intermédiaires préalablement ajoutés. Le deuxième algorithme, TAPE, est la première méthode d’exploration de cavités en 3D qui minimise le temps de mission et la longueur du fil déroulé. En moyenne, le trajet global est 4% plus long que la méthode qui résout le TSP, mais le fil reste sous la longueur autorisée dans 100% des cas simulés, contre 53% avec la méthode initiale. L’approche utilise une architecture hiérarchique à 2 niveaux : la planification globale résout un TSP après extraction des frontières, et la planification locale minimise le coût du chemin et la longueur de fil via une fonction de décision. L’intégration de ces deux outils dans le NetherDrone produit un système intelligent pour l’exploration autonome, doté de fonctionnalités semi-autonomes pour une interaction avec l’opérateur. Les travaux réalisés ouvrent la porte à de nouvelles approches de navigation dans le domaine des missions d’inspection, de cartographie et de recherche et sauvetage

A Dynamical System Approach for Resource-Constrained Mobile Robotics

The revolution of autonomous vehicles has led to the development of robots with abundant sensors, actuators with many degrees of freedom, high-performance computing capabilities, and high-speed communication devices. These robots use a large volume of information from sensors to solve diverse problems. However, this usually leads to a significant modeling burden as well as excessive cost and computational requirements. Furthermore, in some scenarios, sophisticated sensors may not work precisely, the real-time processing power of a robot may be inadequate, the communication among robots may be impeded by natural or adversarial conditions, or the actuation control in a robot may be insubstantial. In these cases, we have to rely on simple robots with limited sensing and actuation, minimal onboard processing, moderate communication, and insufficient memory capacity. This reality motivates us to model simple robots such as bouncing and underactuated robots making use of the dynamical system techniques. In this dissertation, we propose a four-pronged approach for solving tasks in resource-constrained scenarios: 1) Combinatorial filters for bouncing robot localization; 2) Bouncing robot navigation and coverage; 3) Stochastic multi-robot patrolling; and 4) Deployment and planning of underactuated aquatic robots. First, we present a global localization method for a bouncing robot equipped with only a clock and contact sensors. Space-efficient and finite automata-based combinatorial filters are synthesized to solve the localization task by determining the robot’s pose (position and orientation) in its environment. Second, we propose a solution for navigation and coverage tasks using single or multiple bouncing robots. The proposed solution finds a navigation plan for a single bouncing robot from the robot’s initial pose to its goal pose with limited sensing. Probabilistic paths from several policies of the robot are combined artfully so that the actual coverage distribution can become as close as possible to a target coverage distribution. A joint trajectory for multiple bouncing robots to visit all the locations of an environment is incrementally generated. Third, a scalable method is proposed to find stochastic strategies for multi-robot patrolling under an adversarial and communication-constrained environment. Then, we evaluate the vulnerability of our patrolling policies by finding the probability of capturing an adversary for a location in our proposed patrolling scenarios. Finally, a data-driven deployment and planning approach is presented for the underactuated aquatic robots called drifters that creates the generalized flow pattern of the water, develops a Markov-chain based motion model, and studies the long- term behavior of a marine environment from a flow point-of-view. In a broad summary, our dynamical system approach is a unique solution to typical robotic tasks and opens a new paradigm for the modeling of simple robotics system

Reitinsuunnittelu määrätyssä järjestyksessä tehtäville peltotöille usean työkoneen yhteistyönä

Coverage path planning is the task of finding a collision free path that passes over every point of an area or volume of interest. In agriculture, the coverage task is encountered especially in the process of crop cultivation. Several tasks are performed on the field, one after the other, during the cultivation cycle. Cooperation means that multiple agents, in this case vehicles, are working together towards a common goal. Several studies consider the problem where a single task is divided and assigned among the agents. In this thesis, however, the vehicles have different tasks that are sequentially dependent, that is, the first task must be completed before the other. The tasks are performed simultaneously on the same area. The literature review suggests that there is a lack of previous research on this topic. The objective of this thesis was to develop an algorithm to solve the cooperative coverage path planning problem for sequentially dependent tasks. A tool chain that involves Matlab, Simulink and Visual Studio was adapted for the development and testing of the solution. A development and testing architecture was designed including a compatible interface to a simulation and a real-life test environment. Two different algorithms were implemented based on the idea of computing short simultaneous paths at a time and scheduling them in real-time. The results were successfully demonstrated in a real-life test environment with two tractors equipped with a disc cultivator and a seeder. The objective was to sow the test area. The test drives show that with the algorithms that were developed in this thesis it is possible to perform two sequentially dependent agricultural coverage tasks simultaneously on the same area.Kattavassa reitinsuunnittelussa yritetään löytää polku, jonka aikana määritelty ala tai tilavuus tulee käytyä läpi niin että alueen jokainen piste on käsitelty. Maataloudessa tämä tehtävä on merkityksellinen erityisesti peltoviljelyssä. Useita peltotöitä suoritetaan yksi toisensa jälkeen samalla alueella viljelyvuoden aikana. Useissa tutkimuksissa käsitellään yhteistyönä tehtävää reitinsuunnittelua, jossa yksi tehtävä on jaettu osiin ja osat jaetaan useiden tekijöiden kuten robottien kesken. Tässä diplomityössä peltotyökoneilla on kuitenkin omat erilliset tehtävänsä, joilla on määrätty järjestys, eli niiden suorittaminen riippuu työjärjestyksestä. Työkoneet työskentelevät samanaikaisesti samalla alueella. Diplomityössä tehty kirjallisuuskatsaus viittaa siihen, että vastaavaa aihetta ei ole aiemmin tutkittu. Tämän diplomityön tavoitteena on kehittää algoritmi, jolla voidaan toteuttaa reitinsuunnittelu määrätyssä järjestyksessä tehtäville peltotöille usean peltotyökoneen yhteistyönä. Algoritmikehitystä ja testausta varten suunniteltiin yhtenäinen rajapinta, jolla algoritmia voitaisiin testata sekä simulaatiossa että todellisessa testitilanteessa. Algoritmikehityksessä käytettiin työkaluina Matlab, Simulink ja Visual Studio -ohjelmia. Työssä toteutettiin kaksi algoritmia, jotka perustuvat samaan ideaan: suunnitellaan kerrallaan kaksi lyhyttä samanaikaista polkua, jotka ajoitetaan reaaliajassa. Algoritmeja testattiin todellisessa testiympäristössä kahden työkoneen yhteistyönä, kun tavoitteena on kylvää koko testialue. Ensimmäinen työvaihe suoritettiin lautasmuokkaimella ja toinen kylvökoneella. Testiajot osoittavat, että diplomityössä kehitetyillä algoritmeilla voidaan ohjata kahden toisistaan riippuvaisen peltotyön toteutus samanaikaisesti samalla peltoalueella