A survey on robotic technologies for forest firefighting: Applying drone swarms to improve firefighters’ efficiency and safety

Forest firefighting missions encompass multiple tasks related to prevention, surveillance, and extinguishing. This work presents a complete survey of firefighters on the current problems in their work and the potential technological solutions. Additionally, it reviews the efforts performed by the academy and industry to apply different types of robots in the context of firefighting missions. Finally, all this information is used to propose a concept of operation for the comprehensive application of drone swarms in firefighting. The proposed system is a fleet of quadcopters that individually are only able to visit waypoints and use payloads, but collectively can perform tasks of surveillance, mapping, monitoring, etc. Three operator roles are defined, each one with different access to information and functions in the mission: Mission commander, team leaders, and team members. These operators take advantage of virtual and augmented reality interfaces to intuitively get the information of the scenario and, in the case of the mission commander, control the drone swarmThis research received no external fundin

Echo State Learning for Wireless Virtual Reality Resource Allocation in UAV-enabled LTE-U Networks

In this paper, the problem of resource management is studied for a network of wireless virtual reality (VR) users communicating using an unmanned aerial vehicle (UAV)-enabled LTE-U network. In the studied model, the UAVs act as VR control centers that collect tracking information from the VR users over the wireless uplink and, then, send the constructed VR images to the VR users over an LTE-U downlink. Therefore, resource allocation in such a UAV-enabled LTE-U network must jointly consider the uplink and downlink links over both licensed and unlicensed bands. In such a VR setting, the UAVs can dynamically adjust the image quality and format of each VR image to change the data size of each VR image, then meet the delay requirement. Therefore, resource allocation must also take into account the image quality and format. This VR-centric resource allocation problem is formulated as a noncooperative game that enables a joint allocation of licensed and unlicensed spectrum bands, as well as a dynamic adaptation of VR image quality and format. To solve this game, a learning algorithm based on the machine learning tools of echo state networks (ESNs) with leaky integrator neurons is proposed. Unlike conventional ESN based learning algorithms that are suitable for discrete-time systems, the proposed algorithm can dynamically adjust the update speed of the ESN's state and, hence, it can enable the UAVs to learn the continuous dynamics of their associated VR users. Simulation results show that the proposed algorithm achieves up to 14% and 27.1% gains in terms of total VR QoE for all users compared to Q-learning using LTE-U and Q-learning using LTE

improving path planning of unmanned aerial vehicles in an immersive environment using meta-paths and terrain information

Effective command and control of unmanned aerial vehicles (UAVs) is an issue under investigation as the military pushes toward more automation and incorporation of technology into their operational strategy. UAVs require the intelligence to maneuver safely along a path to an intended target while avoiding obstacles such as other aircraft or enemy threats. To date, path-planning algorithms (designed to aid the operator in the control of semi-autonomous UAVs) have been limited to providing only a single solution (alternate path) without utilizing input or feedback from the UAV operator. The work presented in this thesis builds off of and improves an existing path planner. The original path planner presents a unique platform for decision making in a three-dimensional environment where multiple solution paths are generated using Particle Swarm Optimization (PSO) and returned to the operator for evaluation. The paths are optimized to minimize risk due to enemy threats, to minimize fuel consumption incurred by deviating from the original path, and to maximize reconnaissance over predefined targets. The work presented in this thesis focuses on improving the mathematical models of these objectives. Terrain data is also incorporated into the path planner to ensure that the generated alternate paths are feasible and at a safe height above ground. An effective interface is needed to evaluate the alternate paths returned by PSO. A meta-path is a new concept presented in this thesis to address this issue. Meta-paths allow an operator to explore paths in an efficient and organized manner by displaying multiple alternate paths as a single path cloud. The interface was augmented with more detailed information on these paths to allow the operator to make a more informed decision. Two other interaction techniques were investigated to allow the operator more interactive control over the results displayed by the path planner. Viewing the paths in an immersive environment enhances the operator\u27s understanding of the situation and the options while facilitating better decision making. The problem formulation and solution implementation are described along with the results from several simulated scenarios. Preliminary assessments using simulated scenarios show the usefulness of these features in improving command and control of UAVs. Finally, a user study was conducted to gauge how different visualization capabilities affect operator performance when using an interactive path planning tool. The study demonstrates that viewing alternate paths in 3D instead of 2D takes more time because the operator switches between multiple views of the paths but also suggests that 3D is better for allowing the operator to understand more complex situations

Distributed Real-Time Hardware- and Man-in-the-loop Simulation for the ICARO II Unmanned Systems Autopilot

The autopilot market for small and research UAVs offers several products, but most of them, although widely configurable or even open-source, do not constitute a practical and safe development system for custom guidance, navigation and control systems. The ICARO project aims at providing the small UAV community with a valid autopilot alternative. The ICARO autopilot exploits rapid control system prototyping techniques and immersive manned simulation with the possibility of testing the autopilot using the Hardware- In-the-Loop (HIL) approach. This paper describes the hardware-in-the-loop and man-in-the-loop simulator for the ICARO II platform together with the synchronization protocol we developed to keep simulator and autopilot synchronized. Experimental evidence of the effectiveness of the synchronization protocol is given

Effects of Visual Interaction Methods on Simulated Unmanned Aircraft Operator Situational Awareness

The limited field of view of static egocentric visual displays employed in unmanned aircraft controls introduces the soda straw effect on operators, which significantly affects their ability to capture and maintain situational awareness by not depicting peripheral visual data. The problem with insufficient operator situational awareness is the resulting increased potential for error and oversight during operation of unmanned aircraft, leading to accidents and mishaps costing United States taxpayers between $4 million to$54 million per year. The purpose of this quantitative experimental completely randomized design study was to examine and compare use of dynamic eyepoint to static visual interaction in a simulated stationary egocentric environment to determine which, if any, resulted in higher situational awareness. The theoretical framework for the study established the premise that the amount of visual information available could affect the situational awareness of an operator and that increasing visual information through dynamic eyepoint manipulation may result in higher situational awareness than static visualization. Four experimental dynamic visual interaction methods were examined (analog joystick, head tracker, uninterrupted hat/point of view switch, and incremental hat/point of view switch) and compared to a single static method (the control treatment). The five methods were used in experimental testing with 150 participants to determine if the use of a dynamic eyepoint significantly increased the situational awareness of a user within a stationary egocentric environment, indicating that employing dynamic control would reduce the occurrence or consequences of the soda straw effect. The primary difference between the four dynamic visual interaction methods was their unique manipulation approaches to control the pitch and yaw of the simulated eyepoint. The identification of dynamic visual interaction increasing user SA may lead to the further refinement of human-machine-interface (HMI), teleoperation, and unmanned aircraft control principles, with the pursuit and performance of related research

Improving particle swarm optimization path planning through inclusion of flight mechanics

Military engagements are continuing the movement toward automated and unmanned vehicles for a variety of simple and complex tasks. This allows humans to stay away from dangerous situations and use their skills for more difficult tasks. One important piece of this strategy is the use of automated path planners for unmanned aerial vehicles (UAVs). Current UAV operation requires multiple individuals to control a single plane, tying up important human resources. Often paths are planned by creating waypoints for a vehicle to fly through, with the intention of doing reconnaissance while avoiding as much danger to the plane as possible. Path planners often plan routes without taking into consideration the UAV\u27s ability to perform the maneuvers required to fly the specified waypoints, instead relying upon them to fly as close as possible. This thesis presents a path planner solution incorporating vehicle mechanics to insure feasible flight paths. This path planner uses Particle Swarm Optimization (PSO) and digital pheromones to generate multiple three-dimensional flight paths for the operator to choose from. B-spline curves are generated using universal interpolation with each path waypoint representing a control point. The b-spline curve represents the flight path of the UAV. Each point along the curve is evaluated for fuel efficiency, threat avoidance, reconnaissance, terrain avoidance, and vehicle mechanics. Optimization of the flight path occurs based on operator defined performance characteristics, such as maximum threat avoidance or minimum vehicle dynamics cost. These performance characteristics can be defined for each unique aircraft, allowing the same formulation to be used for any aircraft. The vehicle mechanics conditions considered are pull-out, glide, climb, and steady, level, co-ordinate turns. Calculating the flight mechanics requires knowing the velocity and angle of the plane, calculated using the derivative of the point on the curve. The flight mechanics of the path allows the path planner to determine whether the path exceeds the maximum load factor (G-force), minimum velocity (stall velocity), or the minimum turning radius. Comparing the results between PSO Path Planner with flight mechanics and PSO Path Planner without flight mechanics over five scenarios indicates an increase in the feasibility of the returned paths. Visualizing the flight paths was improved by changing the original waypoint based visualization to a b-spline curve representation. Using b-spline curves allows for an accurate representation of the actual UAV flight path especially when considering turns. Operators no longer must create a mental representation of the flight path to match the waypoints

Vision-Based Autonomous Following of a Moving Platform and Landing for an Unmanned Aerial Vehicle

Interest in Unmanned Aerial Vehicles (UAVs) has increased due to their versatility and variety of applications, however their battery life limits their applications. Heterogeneous multi-robot systems can offer a solution to this limitation, by allowing an Unmanned Ground Vehicle (UGV) to serve as a recharging station for the aerial one. Moreover, cooperation between aerial and terrestrial robots allows them to overcome other individual limitations, such as communication link coverage or accessibility, and to solve highly complex tasks, e.g., environment exploration, infrastructure inspection or search and rescue. This work proposes a vision-based approach that enables an aerial robot to autonomously detect, follow, and land on a mobile ground platform. For this purpose, ArUcO fiducial markers are used to estimate the relative pose between the UAV and UGV by processing RGB images provided by a monocular camera on board the UAV. The pose estimation is fed to a trajectory planner and four decoupled controllers to generate speed set-points relative to the UAV. Using a cascade loop strategy, these set-points are then sent to the UAV autopilot for inner loop control. The proposed solution has been tested both in simulation, with a digital twin of a solar farm using ROS, Gazebo and Ardupilot Software-in-the-Loop (SiL); and in the real world at IST Lisbon’s outdoor facilities, with a UAV built on the basis of a DJ550 Hexacopter and a modified Jackal ground robot from DJI and Clearpath Robotics, respectively. Pose estimation, trajectory planning and speed set-point are computed on board the UAV, using a Single Board Computer (SBC) running Ubuntu and ROS, without the need for external infrastructure.This research was funded by the ISR/LARSyS Strategic Funding through the FCT project UIDB/50009/2020, the DURABLE project, under the Interreg Atlantic Area Programme through the European Regional Development Fund (ERDF), the Andalusian project UMA18-FEDERJA-090 and the University of Málaga Research Plan. Partial funding for open access charge: Universidad de Málag