Dual Control for Exploitation and Exploration (DCEE) in Autonomous Search

This paper proposes an optimal autonomous search framework, namely Dual Control for Exploration and Exploitation (DCEE), for a target at unknown location in an unknown environment. Source localisation is to find sources of atmospheric hazardous material release in a partially unknown environment. This paper proposes a control theoretic approach to this autonomous search problem. To cope with an unknown target location, at each step, the target location is estimated by Bayesian inference. Then a control action is taken to minimise the error between future robot position and the hypothesised future estimation of the target location. The latter is generated by hypothesised measurements at the corresponding future robot positions (due to the control action) with the current estimation of the target location as a prior. It shows that this approach can take into account both the error between the next robot position and the estimate of the target location, and the uncertainty of the estimate. This approach is further extended to the case with not only an unknown source location, but also an unknown local environment (e.g. wind speed and direction). Different from current information theoretic approaches, this new control theoretic approach achieves the optimal trade-off between exploitation and exploration in a unknown environment with an unknown target by driving the robot moving towards estimated target location while reducing its estimation uncertainty. This scheme is implemented using particle filtering on a mobile robot. Simulation and experimental studies demonstrate promising performance of the proposed approach. The relationships between the proposed approach, informative path planning, dual control, and classic model predictive control are discussed and compared

Autonomous search of an airborne release in urban environments using informed tree planning

The use of autonomous vehicles for chemical source localisation is a key enabling tool for disaster response teams to safely and efficiently deal with chemical emergencies. Whilst much work has been performed on source localisation using autonomous systems, most previous works have assumed an open environment or employed simplistic obstacle avoidance, separate to the estimation procedure. In this paper, we explore the coupling of the path planning task for both source term estimation and obstacle avoidance in a holistic framework. The proposed system intelligently produces potential gas sampling locations based on the current estimation of the wind field and the local map. Then a tree search is performed to generate paths toward the estimated source location that traverse around any obstacles and still allow for exploration of potentially superior sampling locations. The proposed informed tree planning algorithm is then tested against the Entrotaxis technique in a series of high fidelity simulations. The proposed system is found to reduce source position error far more efficiently than Entrotaxis in a feature rich environment, whilst also exhibiting vastly more consistent and robust results

On the use of autonomous unmanned vehicles in response to hazardous atmospheric release incidents

Recent events have induced a surge of interest in the methods of response to releases of hazardous materials or gases into the atmosphere. In the last decade there has been particular interest in mapping and quantifying emissions for regulatory purposes, emergency response, and environmental monitoring. Examples include: responding to events such as gas leaks, nuclear accidents or chemical, biological or radiological (CBR) accidents or attacks, and even exploring sources of methane emissions on the planet Mars. This thesis presents a review of the potential responses to hazardous releases, which includes source localisation, boundary tracking, mapping and source term estimation. [Continues.]</div

AID-RL: Active information-directed reinforcement learning for autonomous source seeking and estimation

This paper proposes an active information-directed reinforcement learning (AID-RL) framework for autonomous source seeking and estimation problem. Source seeking requires the search agent to move towards the true source, and source estimation demands the agent to maintain and update its knowledge regarding the source properties such as release rate and source position. These two objectives give rise to the newly developed framework, namely, dual control for exploration and exploitation. In this paper, the greedy RL forms an exploitation search strategy that navigates the agent to the source position, while the information-directed search commands the agent to explore most informative positions to reduce belief uncertainty. Extensive results are presented using a high-fidelity dataset for autonomous search, which validates the effectiveness of the proposed AID-RL and highlights the importance of active exploration in improving sampling efficiency and search performance

Concurrent Active Learning in Autonomous Airborne Source Search: Dual Control for Exploration and Exploitation

In this paper, a concurrent learning framework is developed for source search in an unknown environment using autonomous platforms equipped with onboard sensors. Distinct from the existing solutions that require significant computational power for Bayesian estimation and path planning, the proposed solution is computationally affordable for onboard processors. A new concept of concurrent learning using multiple parallel estimators is proposed to learn the operational environment and quantify estimation uncertainty. The search agent is empowered with dual capability of exploiting current estimated parameters to track the source and probing the environment to reduce the impacts of uncertainty, namely Concurrent Learning based Dual Control for Exploration and Exploitation (CL-DCEE). In this setting, the control action not only minimises the tracking error between future agent's position and estimated source location, but also the uncertainty of predicted estimation. More importantly, the rigorous proven properties such as the convergence of CL-DCEE algorithm are established under mild assumptions on noises, and the impact of noises on the search performance is examined. Simulation results are provided to validate the effectiveness of the proposed CL-DCEE algorithm. Compared with the information-theoretic approach, CL-DCEE not only guarantees convergence, but produces better search performance and consumes much less computational time

Reinforcement learning for source location estimation: a multi-step approach

Gas leaks present an undeniable safety concern, the ability to swiftly and accurately detect the source of a leak and pertinent details is critical for effective emergency response. The limited precision of sensors and environmental noise introduce significant uncertainty and randomness, complicating the resolution of such issues. To address these challenges, this study introduces a new approach that integrates multi-step deep reinforcement learning algorithms with Bayesian inference to estimate source information. Compared to single-step Reinforcement Learning and Entrotaxis methods, the multi-step update mechanism in this problem allows the agent to locate sources position more efficiently. This approach not only increases the search’s success rate but also decreases the number of time steps needed for successful detection. Experiments conducted in continuous and discrete environments of equal scale and parameters corroborate the efficiency of our method in tracing the source of gas leaks.<br/

Cooperative Active Learning based Dual Control for Exploration and Exploitation in Autonomous Search

In this paper, a multi-estimator based computationally efficient algorithm is developed for autonomous search in an unknown environment with an unknown source. Different from the existing approaches that require massive computational power to support nonlinear Bayesian estimation and complex decision-making process, an efficient cooperative active learning based dual control for exploration and exploitation (COAL-DCEE) is developed for source estimation and path planning. Multiple cooperative estimators are deployed for environment learning process, which is helpful to improving the search performance and robustness against noisy measurements. The number of estimators used in COAL-DCEE is much smaller than that of particles required for Bayesian estimation in information-theoretic approaches. Consequently, the computational load is significantly reduced. As an important feature of this study, the convergence and performance of COAL-DCEE are established in relation to the characteristics of sensor noises and turbulence disturbances. Numerical and experimental studies have been carried out to verify the effectiveness of the proposed framework. Compared with existing approaches, COAL-DCEE not only provides convergence guarantee, but also yields comparable search performance using much less computational power

????????? ????????? ????????? ???????????? ????????? ?????? ?????? ??? ?????? ??????

Department of Mechanical EngineeringThe potential danger of invisible hazardous substance leakage accident is increasing, such as hazardous chemical leakage accidents in industrial complexes, potential risks of aging nuclear power plants, and international chemical terrorism threats. In particular, hazardous chemical, biological, or radioactive substances leaked into the atmosphere cause irreversible damage to nature, and there is a risk of human damage if prompt action is not taken. Therefore, estimating the emission source and the amount of invisible hazardous substances is required to minimize human casualties and increase public safety. As the risk of hazardous material leakage and potential terrorism increases in random places, it is difficult using traditional systems such as pre-installed ground sensors in a specific area. This thesis proposes autonomous search method for estimating the source of hazardous materials using a mobile sensor attached to an unmanned aerial vehicle (UAV). Since the mobile sensor can be freely deployed to any arbitrary places, it is possible to monitor a wider area with a relatively low cost. Besides, this approach is an unmanned autonomous system, so it has the advantage of minimizing secondary human casualties that may additionally occur during search. The source term estimation (STE) using mobile sensors is considered to be a challenging problem because the sensor measurements from atmospheric gas dispersion are sparse, intermittent, and time-varying due to the turbulence and the sensor noise. Thus, Bayesian inference-based estimation technique, sequential Monte Carlo method (i.e., particle filter), is used to estimate the source by using the inaccurate measurements which is easily influenced by air turbulence and sensor noise in this thesis. The autonomous search algorithms using information theory are also proposed. In the proposed algorithms, the information entropy (i.e., uncertainty of estimation) is calculated by using information theory and the agent choose the action to move to the next sensing location that can minimize the expected uncertainty. In other words, the proposed information-theoretic search algorithm is reward-based decision making approaches that use information entropy as a reward. The receding horizon and Gaussian mixture model clustering approaches are adopted to improve the search performance in various environment. Since the time required to compute all of the respective rewards increases as the number of action candidates increases, the policy-based autonomous source term search and estimation algorithm is proposed using deep neural network and reinforcement learning approach to determine efficient search path considering continuous action space. Furthermore, this thesis proposes a cooperative search method for multiple unmanned mobile vehicles based on game theory. The inaccuracy of sensor measurement values can be reduced by using multiple mobile sensors with the fusion approach, so the source of hazardous substances can be quickly estimated. The negotiation based on the game theory can improve the group search performance for source term estimation and search. Finally, to verify the performance of the proposed algorithm, numerical simulation and flight test results using an actual gas measurement sensor and multicopter drone are presented.ope

On gas source declaration methods for single-robot search

Source declaration, along with plume finding and plume tracking, is one of the needed processes for gas source localization (GSL). It is a fundamental part of the search, since it is responsible to decide whether the gas source has been found, and also to pinpoint its location. Despite its importance, source declaration is often ignored in most of the GSL research, the criteria for termination being selected in a seemingly arbitrary manner, or even not being discussed at all. A clear example of this is the large number of experiments in the literature that are declared concluded whenever the robot manages to physically reach the source, without formally declaring it. In this work, we seek to provide an overview of the most significant declaration methods that have been used in state-of-the-art research for single-robot GSL, analyzing their strengths and weaknesses. We also provide a preliminary experimental validation of these methods, focusing on how stable their performance is when their input parameters are modified.Universidad de Málaga. Campus de Excelencia Internacional Andalucía Tec

Information-based path planning for source term estimation using an unmanned aerial vehicle: algorithms and experiments

Department of Mechanical Enginering (Mechanical Engineering)Searching and estimating source information such as the location and release rate, called a source term, have many applications across environmental, medical, and security domains. For autonomous source search and estimation in a turbulent environment, this thesis presents two information-theoretic search strategies. Firstly, Gaussian mixture model (GMM) based infotaxis, termed as GMM-Infotaxis, is presented. The GMM is used to determine the action candidates for the next best informative sampling position in a continuous domain by appropriately clustering possible source locations obtained from the particle filter, compared with Infotaxis using discrete action candidates. This facilitates the better trade-off between exploitation and exploration for search, resulting in more efficient search and better estimation performance. However, GMM-Infotaxis has limitations in complex environments with many obstacles such as urban area, as this approach only predicts one step ahead action and the obstacles prevent efficient search. To address this problem, Infotaxis combined with the Rapidly-exploring Random Trees (RRT) is proposed and termed as RRT-Infotaxis. By introducing new utility function which is designed to maximize entropy reduction and minimize searching path at the same time, RRT-Infotaxis has advantage of searching efficient path in obstacle-rich environments. With proposed utility function, this approach is designed not only to avoid obstacles but also to sample the next best sampling positions considering several steps ahead in a continuous domain. Numerical simulations for both strategies, GMM-Infotaxis and RRT-Infotaxis, are implemented to prove the enhanced performance compared to the conventional Infotaxis. Numerical simulations show that in an open space the performance of GMM-Infotaxis is better than the conventional Infotaxis and in various urban environments RRT-Infotaxis outperforms both original Infotaxis and GMM-Infotaxis. Besides, real outdoor flight experiments using a multirotor UAV in an open space for GMM-Infotaxis are conducted. It shows the superior performance of the GMM-Infotaxis compared with the original Infotaxis method.ope