Multi-Robot Path Planning Combining Heuristics and Multi-Agent Reinforcement Learning

Multi-robot path finding in dynamic environments is a highly challenging classic problem. In the movement process, robots need to avoid collisions with other moving robots while minimizing their travel distance. Previous methods for this problem either continuously replan paths using heuristic search methods to avoid conflicts or choose appropriate collision avoidance strategies based on learning approaches. The former may result in long travel distances due to frequent replanning, while the latter may have low learning efficiency due to low sample exploration and utilization, and causing high training costs for the model. To address these issues, we propose a path planning method, MAPPOHR, which combines heuristic search, empirical rules, and multi-agent reinforcement learning. The method consists of two layers: a real-time planner based on the multi-agent reinforcement learning algorithm, MAPPO, which embeds empirical rules in the action output layer and reward functions, and a heuristic search planner used to create a global guiding path. During movement, the heuristic search planner replans new paths based on the instructions of the real-time planner. We tested our method in 10 different conflict scenarios. The experiments show that the planning performance of MAPPOHR is better than that of existing learning and heuristic methods. Due to the utilization of empirical knowledge and heuristic search, the learning efficiency of MAPPOHR is higher than that of existing learning methods

Real-time reach planning for animated characters using hardware acceleration

We present a heuristic-based real-time reach planning algorithm for virtual human figures. Given the start and goal positions in a 3D workspace, our problem is to compute a collision-free path that specifies all the configurations for a human arm to move from the start to the goal. Our algorithm consists of three modules: spatial search, inverse kinematics, and collision detection. For the search module, instead of searching in joint configuration space like most existing motion planning methods do, we run a direct search in the workspace, guided by a heuristic distance-to-goal evaluation function. The inverse kinematics module attempts to select natural posture configurations for the arm along the path found in the workspace. During the search, candidate configurations will be checked for collisions taking advantage of the graphics hardware – depth buffer. The algorithm is fast and easy to implement. It allows real-time planning not only in static, structured environments, but also in dynamic, unstructured environments. No preprocessing and prior knowledge about the environment is required. Several examples are shown illustrating the competence of the planner at generating motion plans for a typical human arm model with seven degrees of freedom

Robot motion planning using real-time heuristic search

Autonomous mobile robots must be able to plan quickly and stay reactive to the world around them. Currently, navigating in the presence of dynamic obstacles is a problem that modern techniques struggle to handle in a real-time manner, even when the environment is known. The solutions range from using: 1) sampling-based algorithms which cut down on the shear size of these state spaces, 2) algorithms which quickly try to plan complete paths to the goal (to avoid local minima) and 3) using real-time search techniques designed for static worlds. Each of these methods have fundamental flaws that prevent it from being used in practice. In this thesis I offer three proposed techniques to help improve planning among dynamic obstacles. First, I present a new partitioned learning technique for splitting the costs estimates used by heuristic search techniques into those caused by the static environment and those caused by the dynamic obstacles in the world. This allows for much more accurate learning. Second, I introduce a novel decaying heuristic technique for generalizing cost-to-go over states of the same pose (x. y.theta.v) in the world. Third, I show a garbage collection mechanism for removing useless states from our search to cut down on the overall memory usage. Finally, I present a new algorithm called Partitioned Learning Real-time A*. PLRTA* uses all three of these new enhancements to navigate through worlds with dynamic obstacles in a real-time manner while handling the complex situations in which other algorithms fail. I empirically compare our algorithm to other competing algorithms in a number of random instances as well as hand crafted scenarios designed to highlight desirable behavior in specific situations. I show that PLRTA* outperforms the current state-of-the-art algorithms in terms of minimizing cost over a large number of robot motion planning problems, even when planning in fairly confined environments with up to ten dynamic obstacles

Fuzzy A* for optimum Path Planning in a Large Maze

 Traditional A* path planning, while guaranteeing the shortest path with an admissible heuristic, often employs conservative heuristic functions that neglect potential obstacles and map inaccuracies. This can lead to inefficient searches and increased memory usage in complex environments. To address this, machine learning methods have been explored to predict cost functions, reducing memory load while maintaining optimal solutions. However, these require extensive data collection and struggle in novel, intricate environments. We propose the Fuzzy A* algorithm, an enhancement of the classic A* method, incorporating a new determinant variable to adjust heuristic cost calculations. This adjustment modulates the scope of scanned vertices during searches, optimizing memory usage and computational efficiency. In our approach, unlike traditional A* heuristics that overlook environmental complexities, the Fuzzy A* employs a dynamic heuristic function. This function, leveraging fuzzy logic principles, adapts to varying levels of environmental complexity, allowing a more nuanced estimation of the path cost that considers potential obstructions and route feasibility. This adaptability contrasts with standard machine learning-based solutions, which, while effective in known environments, often falter in unfamiliar or highly complex settings due to their reliance on pre-existing datasets. Our experimental framework involved 100 maze-solving trials in diverse maze configurations, ranging from simple to highly intricate layouts, to evaluate the effectiveness of Fuzzy A*. We employed specific metrics such as path length, computational time, and memory usage for a comprehensive assessment. The results showcased that Fuzzy A* consistently found the shortest paths (99.96% success rate) and significantly reduced memory usage by 67% and 59% compared to Breadth-First-Search (BFS) and traditional A*, respectively. These findings underline the effectiveness of our modified heuristic approach in diverse and challenging environments, highlighting its potential for real-world pathfinding applications

Static and Dynamic Path Planning Using Incremental Heuristic Search

Path planning is an important component in any highly automated vehicle system. In this report, the general problem of path planning is considered first in partially known static environments where only static obstacles are present but the layout of the environment is changing as the agent acquires new information. Attention is then given to the problem of path planning in dynamic environments where there are moving obstacles in addition to the static ones. Specifically, a 2D car-like agent traversing in a 2D environment was considered. It was found that the traditional configuration-time space approach is unsuitable for producing trajectories consistent with the dynamic constraints of a car. A novel scheme is then suggested where the state space is 4D consisting of position, speed and time but the search is done in the 3D space composed by position and speed. Simulation tests shows that the new scheme is capable of efficiently producing trajectories respecting the dynamic constraint of a car-like agent with a bound on their optimality.Comment: Internship Repor

Dynamically Pruned A* for Re-planning in Navigation Meshes

Modern simulations feature crowds of AI-controlled agents moving through dynamic environments, with obstacles appearing or disappearing at run-time. A dynamic navigation mesh can represent the traversable space of such environments. The A* algorithm computes optimal paths through the dual graph of this mesh. When an obstacle is inserted or removed, the mesh changes and agents should re-plan their paths. Many existing re-planning algorithms are too memory-intensive for crowds, or they cannot easily be used on graphs that structurally change. In this paper, we present Dynamically Pruned A* (DPA*), an extension of A* for re-planning optimal paths in dynamic navigation meshes. DPA* has similarities to adaptive algorithms that make the A* heuristic more informed based on previous queries. However, DPA* prunes the search using only the previous path and its relation to the dynamic event. We describe this relation using four scenarios; DPA* uses different rules in each scenario. Our algorithm is memory-friendly and robust against structural changes, which makes it suitable for crowds in dynamic navigation meshes. Experiments show that DPA* performs particularly well in large environments and when the dynamic event is visible to the agent. We integrate the algorithm into crowd simulation software to model large crowds in dynamic environments in real-time