Multi-Robot Path Planning Combining Heuristics and Multi-Agent Reinforcement Learning

Multi-robot path finding in dynamic environments is a highly challenging classic problem. In the movement process, robots need to avoid collisions with other moving robots while minimizing their travel distance. Previous methods for this problem either continuously replan paths using heuristic search methods to avoid conflicts or choose appropriate collision avoidance strategies based on learning approaches. The former may result in long travel distances due to frequent replanning, while the latter may have low learning efficiency due to low sample exploration and utilization, and causing high training costs for the model. To address these issues, we propose a path planning method, MAPPOHR, which combines heuristic search, empirical rules, and multi-agent reinforcement learning. The method consists of two layers: a real-time planner based on the multi-agent reinforcement learning algorithm, MAPPO, which embeds empirical rules in the action output layer and reward functions, and a heuristic search planner used to create a global guiding path. During movement, the heuristic search planner replans new paths based on the instructions of the real-time planner. We tested our method in 10 different conflict scenarios. The experiments show that the planning performance of MAPPOHR is better than that of existing learning and heuristic methods. Due to the utilization of empirical knowledge and heuristic search, the learning efficiency of MAPPOHR is higher than that of existing learning methods

Detecting outlying subspaces for high-dimensional data: the new task, algorithms and performance

[Abstract]: In this paper, we identify a new task for studying the outlying degree (OD) of high-dimensional data, i.e. finding the subspaces (subsets of features) in which the given points are outliers, which are called their outlying subspaces. Since the state-of-the-art outlier detection techniques fail to handle this new problem, we propose a novel detection algorithm, called High-Dimension Outlying subspace Detection (HighDOD), to detect the outlying subspaces of high-dimensional data efficiently. The intuitive idea of HighDOD is that we measure the OD of the point using the sum of distances between this point and its k nearest neighbors. Two heuristic pruning strategies are proposed to realize fast pruning in the subspace search and an efficient dynamic subspace search method with a sample-based learning process has been implemented. Experimental results show that HighDOD is efficient and outperforms other searching alternatives such as the naive top–down, bottom–up and random search methods, and the existing outlier detection methods cannot fulfill this new task effectively

Detecting outlying subspaces for high-dimensional data: a heuristic search approach

[Abstract]: In this paper, we identify a new task for studying the out-lying degree of high-dimensional data, i.e. finding the sub-spaces (subset of features) in which given points are out-liers, and propose a novel detection algorithm, called High-D Outlying subspace Detection (HighDOD). We measure the outlying degree of the point using the sum of distances between this point and its k nearest neighbors. Heuristic pruning strategies are proposed to realize fast pruning in the subspace search and an efficient dynamic subspace search method with a sample-based learning process has been im- plemented. Experimental results show that HighDOD is efficient and outperforms other searching alternatives such as the naive top-down, bottom-up and random search methods. Points in these sparse subspaces are assumed to be the outliers. While knowing which data points are the outliers can be useful, in many applications, it is more important to identify the subspaces in which a given point is an outlier, which motivates the proposal of a new technique in this paper to handle this new task

Time preference and decision rules in a price search experiment

Structural econometric methods that assume agents have rational expectations are often criticized. Yet, little is known about the relative costs and benefits of adopting alternative empirical strategies. This paper compares three procedures for inference about a single structural parameter using data from a laboratory price search experiment. Our novel experimental design induces preferences up to the subjective rate of time preference, leaving unrestricted only this parameter and the decision rule that subjects use in solving the search task. We analyze the experimental data under the assumptions of both rational expectations and heuristic behavior, and we also draw inferences using a simple revealed preference analysis that does not require strong behavioral assumptions. We find that the revealed-preference analysis does not provide much information about the discount rate, while the two specifications with stronger behavioral assumptions provide sharper and statistically identical inferences about the population's discount rate distribution. However, substantial differences in inference appear at the individual level. We compare the individual discount-rate estimates to an external measure of forward looking behavior obtained for each subject using an instrument validated in the psychology literature. The estimates obtained under heuristic behavior are statistically significantly positively correlated with our external measure of time preference, while the estimates obtained under rational expectations and the revealed-preference estimates are not.

Reinforcement Learning for Mutation Operator Selection in Automated Program Repair

Automated program repair techniques aim to aid software developers with the challenging task of fixing bugs. In heuristic-based program repair, a search space of program variants is created by applying mutation operations on the source code to find potential patches for bugs. Most commonly, every selection of a mutation operator during search is performed uniformly at random. The inefficiency of this critical step in the search creates many variants that do not compile or break intended functionality, wasting considerable resources as a result. In this paper, we address this issue and propose a reinforcement learning-based approach to optimise the selection of mutation operators in heuristic-based program repair. Our solution is programming language, granularity-level, and search strategy agnostic and allows for easy augmentation into existing heuristic-based repair tools. We conduct extensive experimentation on four operator selection techniques, two reward types, two credit assignment strategies, two integration methods, and three sets of mutation operators using 22,300 independent repair attempts. We evaluate our approach on 353 real-world bugs from the Defects4J benchmark. Results show that the epsilon-greedy multi-armed bandit algorithm with average credit assignment is best for mutation operator selection. Our approach exhibits a 17.3% improvement upon the baseline, by generating patches for 9 additional bugs for a total of 61 patched bugs in the Defects4J benchmark

An investigation into Off-Link IPv6 host enumeration search methods

This research investigated search methods for enumerating networked devices on off-link 64 bit Internet Protocol version 6 (IPv6) subnetworks. IPv6 host enumeration is an emerging research area involving strategies to enable detection of networked devices on IPv6 networks. Host enumeration is an integral component in vulnerability assessments (VAs), and can be used to strengthen the security profile of a system. Recently, host enumeration has been applied to Internet-wide VAs in an effort to detect devices that are vulnerable to specific threats. These host enumeration exercises rely on the fact that the existing Internet Protocol version 4 (IPv4) can be exhaustively enumerated in less than an hour. The same is not true for IPv6, which would take over 584,940 years to enumerate a single network. As such, research is required to determine appropriate host enumeration search methods for IPv6, given that the protocol is seeing increase global usage. For this study, a survey of Internet resources was conducted to gather information about the nature of IPv6 usage in real-world scenarios. The collected survey data revealed patterns in the usage of IPv6 that influenced search techniques. The research tested the efficacy of various searching algorithms against IPv6 datasets through the use of simulation. Multiple algorithms were devised to test different approaches to host enumeration against 64 bit IPv6 subnetworks. Of these, a novel adaptive heuristic search algorithm, a genetic algorithm and a stripe search algorithm were chosen to conduct off-link IPv6 host enumeration. The suitability of a linear algorithm, a Monte Carlo algorithm and a pattern heuristics algorithm were also tested for their suitability in searching off-link IPv6 networks. These algorithms were applied to two test IPv6 address datasets, one comprised of unique IPv6 data observed during the survey phase, and one comprised of unique IPv6 data generated using pseudorandom number generators. Searching against the two unique datasets was performed in order to determine appropriate strategies for off-link host enumeration under circumstances where networked devices were configured with addresses that represented real-word IPv6 addresses, and where device addresses were configured through some randomisation function. Whilst the outcomes of this research support that an exhaustive enumeration of an IPv6 network is infeasible, it has been demonstrated that devices on IPv6 networks can be enumerated. In particular, it was identified that the linear search technique and the variants tested in this study (pattern search and stripe search), remained the most consistent means of enumerating an IPv6 network. Machine learning methods were also successfully applied to the problem. It was determined that the novel adaptive heuristic search algorithm was an appropriate candidate for search operations. The adaptive heuristic search algorithm successfully enumerated over 24% of the available devices on the dataset that was crafted from surveyed IPv6 address data. Moreover, it was confirmed that stochastic address generation can reduce the effectiveness of enumeration strategies, as all of the algorithms failed to enumerate more than 1% of hosts against a pseudorandomly generated dataset. This research highlights a requirement for effective IPv6 host enumeration algorithms, and presents and validates appropriate methods. The methods presented in this thesis can help to influence the tools and utilities that are used to conduct host enumeration exercises