Learning in Real-Time Search: A Unifying Framework

Real-time search methods are suited for tasks in which the agent is interacting with an initially unknown environment in real time. In such simultaneous planning and learning problems, the agent has to select its actions in a limited amount of time, while sensing only a local part of the environment centered at the agents current location. Real-time heuristic search agents select actions using a limited lookahead search and evaluating the frontier states with a heuristic function. Over repeated experiences, they refine heuristic values of states to avoid infinite loops and to converge to better solutions. The wide spread of such settings in autonomous software and hardware agents has led to an explosion of real-time search algorithms over the last two decades. Not only is a potential user confronted with a hodgepodge of algorithms, but he also faces the choice of control parameters they use. In this paper we address both problems. The first contribution is an introduction of a simple three-parameter framework (named LRTS) which extracts the core ideas behind many existing algorithms. We then prove that LRTA*, epsilon-LRTA*, SLA*, and gamma-Trap algorithms are special cases of our framework. Thus, they are unified and extended with additional features. Second, we prove completeness and convergence of any algorithm covered by the LRTS framework. Third, we prove several upper-bounds relating the control parameters and solution quality. Finally, we analyze the influence of the three control parameters empirically in the realistic scalable domains of real-time navigation on initially unknown maps from a commercial role-playing game as well as routing in ad hoc sensor networks

A Heuristic Method for Task Selection in Persistent ISR Missions Using Autonomous Unmanned Aerial Vehicles

The Persistent Intelligence, Surveillance, and Reconnaissance (PISR) problem seeks to provide timely collection and delivery of data from prioritized ISR tasks using an autonomous Unmanned Aerial Vehicle (UAV). In the literature, PISR is classified as a type of Vehicle Routing Problem (VRP), often called by other names such as persistent monitoring, persistent surveillance, and patrolling. The objective of PISR is to minimize the weighted revisit time to each task (called weighted latency) using an optimal task selection algorithm. In this research, we utilize the average weighted latency as our performance metric and investigate a method for task selection called the Maximal Distance Discounted and Weighted Revisit Period (MD2WRP) utility function. The MD2WRP function is a heuristic method of task selection that uses n+1 parameters, where n is the number of PISR tasks. We develop a two-step optimization method for the MD2WRP parameters to deliver optimal latency performance for any given task configuration, which accommodates both single and multi-vehicle scenarios. To validate our optimization method, we compare the performance of MD2WRP to common Traveling Salesman Problem (TSP) methods for PISR using different task configurations. We find that the optimized MD2WRP function is competitive with the TSP methods, and that MD2WRP often results in steady-state task visit sequences that are equivalent to the TSP solution for a single vehicle. We also compare MD2WRP to other utility methods from the literature, finding thatMD2WRP performs on par with or better than these other methods even when optimizing only one of its n + 1 parameters. To address real-world operational factors, we test MD2WRP with Dubins constraints, no-y zones in the operational area, return-to-base requirements, and the addition and removal of vehicles and tasks mid-mission. For each operational factor, we demonstrate its effect on PISR task selections using MD2WRP and how MD2WRP needs to be modified, if at all, to compensate. Finally, we make practical suggestions about implementing MD2WRP for flight testing, outline potential areas for future study, and offer recommendations about the conduct of PISR missions in general

The delivery dispatching problem with time windows for urban consolidation centers

This paper addresses the dispatch decision problem faced by an urban consolidation center. The center receives orders according to a stochastic arrival process, and dispatches them for the last-mile distribution in batches. The operator of the center aims to fi nd the cost-minimizing consolidation policy, depending on the orders at hand, pre-announced orders, and stochastic arrivals. We present this problem as a variant of the Delivery Dispatching Problem that includes dispatch windows, and model it as a Markov decision problem. For toy-sized instances, we solve this model to optimality. Through numerical experiments on these instances, we show that we approximate the optimal values with an error of less than 2%. Larger instances suff er from intractably large state-, outcome-, and action spaces. We propose an Approximate Dynamic Programming (ADP) algorithm that can handle such instances, using value function approximation to estimate the downstream costs. To cope with large action spaces - with sizes up to 2120 in our experiments - we formulate an integer linear program to be used within our ADP algorithm. To evaluate the performance of our ADP policies, we test against various benchmark policies, including a lookahead policy based on scenario sampling. We test the performance of ADP on a variety of networks. When the dispatching problem provides su fficient fl+6exibility in dispatch times, ADP outperforms our myopic benchmark policies by more than 15%, and lookahead policies by over 10%

A Genetic Algorithm for UAV Routing Integrated with a Parallel Swarm Simulation

This research investigation addresses the problem of routing and simulating swarms of UAVs. Sorties are modeled as instantiations of the NP-Complete Vehicle Routing Problem, and this work uses genetic algorithms (GAs) to provide a fast and robust algorithm for a priori and dynamic routing applications. Swarms of UAVs are modeled based on extensions of Reynolds\u27 swarm research and are simulated on a Beowulf cluster as a parallel computing application using the Synchronous Environment for Emulation and Discrete Event Simulation (SPEEDES). In a test suite, standard measures such as benchmark problems, best published results, and parallel metrics are used as performance measures. The GA consistently provides efficient and effective results for a variety of VRP benchmarks. Analysis of the solution quality over time verifies that the GA exponentially improves solution quality and is robust to changing search landscapes - making it an ideal tool for employment in UAV routing applications. Parallel computing metrics calculated from the results of a PDES show that consistent speedup (almost linear in many cases) can be obtained using SPEEDES as the communication library for this UAV routing application. Results from the routing application and parallel simulation are synthesized to produce a more advanced model for routing UAVs

A fast and reliable broadcast service for LTE-advanced exploiting multihop device-to-device transmissions

Several applications, from the Internet of Things for smart cities to those for vehicular networks, need fast and reliable proximity-based broadcast communications, i.e., the ability to reach all peers in a geographical neighborhood around the originator of a message, as well as ubiquitous connectivity. In this paper, we point out the inherent limitations of the LTE (Long-Term Evolution) cellular network, which make it difficult, if possible at all, to engineer such a service using traditional infrastructure-based communications. We argue, instead, that network-controlled device-to-device (D2D) communications, relayed in a multihop fashion, can efficiently support this service. To substantiate the above claim, we design a proximity-based broadcast service which exploits multihop D2D. We discuss the relevant issues both at the UE (User Equipment), which has to run applications, and within the network (i.e., at the eNodeBs), where suitable resource allocation schemes have to be enforced. We evaluate the performance of a multihop D2D broadcasting using system-level simulations, and demonstrate that it is fast, reliable and economical from a resource consumption standpoint