STDPG: A Spatio-Temporal Deterministic Policy Gradient Agent for Dynamic Routing in SDN

Dynamic routing in software-defined networking (SDN) can be viewed as a centralized decision-making problem. Most of the existing deep reinforcement learning (DRL) agents can address it, thanks to the deep neural network (DNN)incorporated. However, fully-connected feed-forward neural network (FFNN) is usually adopted, where spatial correlation and temporal variation of traffic flows are ignored. This drawback usually leads to significantly high computational complexity due to large number of training parameters. To overcome this problem, we propose a novel model-free framework for dynamic routing in SDN, which is referred to as spatio-temporal deterministic policy gradient (STDPG) agent. Both the actor and critic networks are based on identical DNN structure, where a combination of convolutional neural network (CNN) and long short-term memory network (LSTM) with temporal attention mechanism, CNN-LSTM-TAM, is devised. By efficiently exploiting spatial and temporal features, CNNLSTM-TAM helps the STDPG agent learn better from the experience transitions. Furthermore, we employ the prioritized experience replay (PER) method to accelerate the convergence of model training. The experimental results show that STDPG can automatically adapt for current network environment and achieve robust convergence. Compared with a number state-ofthe-art DRL agents, STDPG achieves better routing solutions in terms of the average end-to-end delay.Comment: 6 pages,5 figures,accepted by IEEE ICC 202

????????? ????????? ????????? ???????????? ????????? ?????? ?????? ??? ?????? ??????

Department of Mechanical EngineeringThe potential danger of invisible hazardous substance leakage accident is increasing, such as hazardous chemical leakage accidents in industrial complexes, potential risks of aging nuclear power plants, and international chemical terrorism threats. In particular, hazardous chemical, biological, or radioactive substances leaked into the atmosphere cause irreversible damage to nature, and there is a risk of human damage if prompt action is not taken. Therefore, estimating the emission source and the amount of invisible hazardous substances is required to minimize human casualties and increase public safety. As the risk of hazardous material leakage and potential terrorism increases in random places, it is difficult using traditional systems such as pre-installed ground sensors in a specific area. This thesis proposes autonomous search method for estimating the source of hazardous materials using a mobile sensor attached to an unmanned aerial vehicle (UAV). Since the mobile sensor can be freely deployed to any arbitrary places, it is possible to monitor a wider area with a relatively low cost. Besides, this approach is an unmanned autonomous system, so it has the advantage of minimizing secondary human casualties that may additionally occur during search. The source term estimation (STE) using mobile sensors is considered to be a challenging problem because the sensor measurements from atmospheric gas dispersion are sparse, intermittent, and time-varying due to the turbulence and the sensor noise. Thus, Bayesian inference-based estimation technique, sequential Monte Carlo method (i.e., particle filter), is used to estimate the source by using the inaccurate measurements which is easily influenced by air turbulence and sensor noise in this thesis. The autonomous search algorithms using information theory are also proposed. In the proposed algorithms, the information entropy (i.e., uncertainty of estimation) is calculated by using information theory and the agent choose the action to move to the next sensing location that can minimize the expected uncertainty. In other words, the proposed information-theoretic search algorithm is reward-based decision making approaches that use information entropy as a reward. The receding horizon and Gaussian mixture model clustering approaches are adopted to improve the search performance in various environment. Since the time required to compute all of the respective rewards increases as the number of action candidates increases, the policy-based autonomous source term search and estimation algorithm is proposed using deep neural network and reinforcement learning approach to determine efficient search path considering continuous action space. Furthermore, this thesis proposes a cooperative search method for multiple unmanned mobile vehicles based on game theory. The inaccuracy of sensor measurement values can be reduced by using multiple mobile sensors with the fusion approach, so the source of hazardous substances can be quickly estimated. The negotiation based on the game theory can improve the group search performance for source term estimation and search. Finally, to verify the performance of the proposed algorithm, numerical simulation and flight test results using an actual gas measurement sensor and multicopter drone are presented.ope