Generalised Entropy MDPs and Minimax Regret

Bayesian methods suffer from the problem of how to specify prior beliefs. One interesting idea is to consider worst-case priors. This requires solving a stochastic zero-sum game. In this paper, we extend well-known results from bandit theory in order to discover minimax-Bayes policies and discuss when they are practical.Comment: 7 pages, NIPS workshop "From bad models to good policies

Efficient RL with Impaired Observability: Learning to Act with Delayed and Missing State Observations

In real-world reinforcement learning (RL) systems, various forms of impaired observability can complicate matters. These situations arise when an agent is unable to observe the most recent state of the system due to latency or lossy channels, yet the agent must still make real-time decisions. This paper introduces a theoretical investigation into efficient RL in control systems where agents must act with delayed and missing state observations. We establish near-optimal regret bounds, of the form $\tilde{\mathcal{O}}(\sqrt{{\rm poly}(H) SAK})$, for RL in both the delayed and missing observation settings. Despite impaired observability posing significant challenges to the policy class and planning, our results demonstrate that learning remains efficient, with the regret bound optimally depending on the state-action size of the original system. Additionally, we provide a characterization of the performance of the optimal policy under impaired observability, comparing it to the optimal value obtained with full observability

복잡하고 불확실한 환경에서 최적 의사 결정을 위한 효율적인 로봇 학습

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·정보공학부, 2021. 2. Songhwai Oh.The problem of sequential decision making under an uncertain and complex environment is a long-standing challenging problem in robotics. In this thesis, we focus on learning a policy function of robotic systems for sequential decision making under which is called a robot learning framework. In particular, we are interested in reducing the sample complexity of the robot learning framework. Hence, we develop three sample efficient robot learning frameworks. The first one is the maximum entropy reinforcement learning. The second one is a perturbation-based exploration. The last one is learning from demonstrations with mixed qualities. For maximum entropy reinforcement learning, we employ a generalized Tsallis entropy regularization as an efficient exploration method. Tsallis entropy generalizes Shannon-Gibbs entropy by introducing a entropic index. By changing an entropic index, we can control the sparsity and multi-modality of policy. Based on this fact, we first propose a sparse Markov decision process (sparse MDP) which induces a sparse and multi-modal optimal policy distribution. In this MDP, the sparse entropy, which is a special case of Tsallis entropy, is employed as a policy regularization. We first analyze the optimality condition of a sparse MDP. Then, we propose dynamic programming methods for the sparse MDP and prove their convergence and optimality. We also show that the performance error of a sparse MDP has a constant bound, while the error of a soft MDP increases logarithmically with respect to the number of actions, where this performance error is caused by the introduced regularization term. Furthermore, we generalize sparse MDPs to a new class of entropy-regularized Markov decision processes (MDPs), which will be referred to as Tsallis MDPs, and analyzes different types of optimal policies with interesting properties related to the stochasticity of the optimal policy by controlling the entropic index. Furthermore, we also develop perturbation based exploration methods to handle heavy-tailed noises. In many robot learning problems, a learning signal is often corrupted by noises such as sub-Gaussian noise or heavy-tailed noise. While most of the exploration strategies have been analyzed under sub-Gaussian noise assumption, there exist few methods for handling such heavy-tailed rewards. Hence, to overcome heavy-tailed noise, we consider stochastic multi-armed bandits with heavy-tailed rewards. First, we propose a novel robust estimator that does not require prior information about a noise distribution, while other existing robust estimators demand prior knowledge. Then, we show that an error probability of the proposed estimator decays exponentially fast. Using this estimator, we propose a perturbation-based exploration strategy and develop a generalized regret analysis scheme that provides upper and lower regret bounds by revealing the relationship between the regret and the cumulative density function of the perturbation. From the proposed analysis scheme, we obtain gap-dependent and gap-independent upper and lower regret bounds of various perturbations. We also find the optimal hyperparameters for each perturbation, which can achieve the minimax optimal regret bound with respect to total rounds. For learning from demonstrations with mixed qualities, we develop a novel inverse reinforcement learning framework using leveraged Gaussian processes (LGP) which can handle negative demonstrations. In LGP, the correlation between two Gaussian processes is captured by a leveraged kernel function. By using properties, the proposed inverse reinforcement learning algorithm can learn from both positive and negative demonstrations. While most existing inverse reinforcement learning (IRL) methods suffer from the lack of information near low reward regions, the proposed method alleviates this issue by incorporating negative demonstrations. To mathematically formulate negative demonstrations, we introduce a novel generative model which can generate both positive and negative demonstrations using a parameter, called proficiency. Moreover, since we represent a reward function using a leveraged Gaussian process which can model a nonlinear function, the proposed method can effectively estimate the structure of a nonlinear reward function.본 학위 논문에서는 시범과 보상함수를 기반으로한 로봇 학습 문제를 다룬다. 로봇 학습 방법은 불확실하고 복잡 업무를 잘 수행 할 수 있는 최적의 정책 함수를 찾는 것을 목표로 한다. 로봇 학습 분야의 다양한 문제 중에, 샘플 복잡도를 줄이는 것에 집중한다. 특히, 효율적인 탐색 방법과 혼합 시범으로 부터의 학습 기법을 개발하여 적은 수의 샘플로도 높은 효율을 갖는 정책 함수를 학습하는 것이 목표이다. 효율적인 탐색 방법을 개발하기 위해서, 우리는 일반화된 쌀리스 엔트로피를 사용한다. 쌀리스 엔트로피는 샤논-깁스 엔트로피를 일반화한 개념으로 엔트로픽 인덱스라는 새로운 파라미터를 도입한다. 엔트로픽 인덱스를 조절함에 따라 다양한 형태의 엔트로피를 만들어 낼 수 있고 각 엔트로피는 서로 다른 레귤러라이제이션 효과를 보인다. 이 성질을 기반으로, 스파스 마르코프 결정과정을 제안한다. 스파스 마르코프 결정과정은 스파스 쌀리스 엔트로피를 이용하여 희소하면서 동시에 다모드의 정책 분포를 표현하는데 효과적이다. 이를 통해서 샤논-깁스 엔트로피를 사용하였을때에 비해 더 좋은 성능을 갖음을 수학적으로 증명하였다. 또한 스파스 쌀리스 엔트로피로 인한 성능 저하를 이론적으로 계산하였다. 스파스 마르코프 결정과정을 더욱 일반화시켜 일반화된 쌀리스 엔트로피 결정과정을 제안하였다. 마찬가지로 쌀리스 엔트로피를 마르코프 결정과정에 추가함으로써 생기는 최적 정책함수의 변화와 성능 저하를 수학적으로 증명하였다. 나아가, 성능저하를 없앨 수 있는 방법인 엔트로픽 인덱스 스케쥴링을 제안하였고 실험적으로 최적의 성능을 갖음을 보였다. 또한, 헤비테일드 잡음이 있는 학습 문제를 해결하기 위해서 외란(Perturbation)을 이용한 탐색 기법을 개발하였다. 로봇 학습의 많은 문제는 잡음의 영향이 존재한다. 학습 신호안에 다양한 형태로 잡음이 들어있는 경우가 있고 이러한 경우에 잡음을 제거 하면서 최적의 행동을 찾는 문제는 효율적인 탐사 기법을 필요로 한다. 기존의 방법론들은 서브 가우시안(sub-Gaussian) 잡음에만 적용 가능했다면, 본 학위 논문에서 제안한 방식은 헤비테일드 잡음을 해결 할 수 있다는 점에서 기존의 방법론들보다 장점을 갖는다. 먼저, 일반적인 외란에 대해서 리그렛 바운드를 증명하였고 외란의 누적분포함수(CDF)와 리그렛 사이의 관계를 증명하였다. 이 관계를 이용하여 다양한 외란 분포의 리그렛 바운드를 계산 가능하게 하였고 다양한 분포들의 가장 효율적인 탐색 파라미터를 계산하였다. 혼합시범으로 부터의 학습 기법을 개발하기 위해서, 오시범을 다룰 수 있는 새로운 형태의 가우시안 프로세스 회귀분석 방식을 개발하였고, 이 방식을 확장하여 레버리지 가우시안 프로세스 역강화학습 기법을 개발하였다. 개발된 기법에서는 정시범으로부터 무엇을 해야 하는지와 오시범으로부터 무엇을 하면 안되는지를 모두 학습할 수 있다. 기존의 방법에서는 쓰일 수 없었던 오시범을 사용 할 수 있게 만듦으로써 샘플 복잡도를 줄일 수 있었고 정제된 데이터를 수집하지 않아도 된다는 점에서 큰 장점을 갖음을 실험적으로 보였다.1 Introduction 1 1.1 Motivation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 1.2 Organization of the Dissertation . . . . . . . . . . . . . . . . . . . 4 2 Background 5 2.1 Learning from Rewards . . . . . . . . . . . . . . . . . . . . . . . . 6 2.1.1 Multi-Armed Bandits . . . . . . . . . . . . . . . . . . . . . 7 2.1.2 Contextual Multi-Armed Bandits . . . . . . . . . . . . . . . 7 2.1.3 Markov Decision Processes . . . . . . . . . . . . . . . . . . 9 2.1.4 Soft Markov Decision Processes . . . . . . . . . . . . . . . . 10 2.2 Learning from Demonstrations . . . . . . . . . . . . . . . . . . . . 12 2.2.1 Behavior Cloning . . . . . . . . . . . . . . . . . . . . . . . . 12 2.2.2 Inverse Reinforcement Learning . . . . . . . . . . . . . . . . 13 3 Sparse Policy Learning 19 3.1 Sparse Policy Learning for Reinforcement Learning . . . . . . . . . 19 3.1.1 Sparse Markov Decision Processes . . . . . . . . . . . . . . 23 3.1.2 Sparse Value Iteration . . . . . . . . . . . . . . . . . . . . . 29 3.1.3 Performance Error Bounds for Sparse Value Iteration . . . 30 3.1.4 Sparse Exploration and Update Rule for Sparse Deep QLearning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 3.1.5 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 34 3.1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 3.2 Sparse Policy Learning for Imitation Learning . . . . . . . . . . . . 46 3.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 48 3.2.2 Principle of Maximum Causal Tsallis Entropy . . . . . . . . 50 3.2.3 Maximum Causal Tsallis Entropy Imitation Learning . . . 54 3.2.4 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 58 3.2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 62 4 Entropy-based Exploration 65 4.1 Generalized Tsallis Entropy Reinforcement Learning . . . . . . . . 65 4.1.1 Maximum Generalized Tsallis Entropy in MDPs . . . . . . 71 4.1.2 Dynamic Programming for Tsallis MDPs . . . . . . . . . . 74 4.1.3 Tsallis Actor Critic for Model-Free RL . . . . . . . . . . . . 78 4.1.4 Experiments Setup . . . . . . . . . . . . . . . . . . . . . . . 79 4.1.5 Experimental Results . . . . . . . . . . . . . . . . . . . . . 84 4.1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92 4.2 E cient Exploration for Robotic Grasping . . . . . . . . . . . . . . 92 4.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 94 4.2.2 Shannon Entropy Regularized Neural Contextual Bandit Algorithm . . . . . . . . . . . . . . . . . . . . . . . . . . . . 99 4.2.3 Theoretical Analysis . . . . . . . . . . . . . . . . . . . . . . 99 4.2.4 Experimental Results . . . . . . . . . . . . . . . . . . . . . 104 4.2.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109 5 Perturbation-Based Exploration 113 5.1 Perturbed Exploration for sub-Gaussian Rewards . . . . . . . . . . 115 5.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 115 5.1.2 Heavy-Tailed Perturbations . . . . . . . . . . . . . . . . . . 117 5.1.3 Adaptively Perturbed Exploration . . . . . . . . . . . . . . 119 5.1.4 General Regret Bound for Sub-Gaussian Rewards . . . . . . 120 5.1.5 Regret Bounds for Speci c Perturbations with sub-Gaussian Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123 5.1.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 127 5.2 Perturbed Exploration for Heavy-Tailed Rewards . . . . . . . . . . 128 5.2.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 130 5.2.2 Sub-Optimality of Robust Upper Con dence Bounds . . . . 132 5.2.3 Adaptively Perturbed Exploration with A p-Robust Estimator . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 133 5.2.4 General Regret Bound for Heavy-Tailed Rewards . . . . . . 136 5.2.5 Regret Bounds for Speci c Perturbations with Heavy-Tailed Rewards . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 139 5.2.6 Experiments . . . . . . . . . . . . . . . . . . . . . . . . . . 144 5.2.7 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 148 6 Inverse Reinforcement Learning with Negative Demonstrations149 6.1 Leveraged Gaussian Processes Inverse Reinforcement Learning . . 151 6.1.1 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . 152 6.1.2 Background . . . . . . . . . . . . . . . . . . . . . . . . . . . 155 6.1.3 Gaussian Process Regression . . . . . . . . . . . . . . . . . 156 6.1.4 Leveraged Gaussian Processes . . . . . . . . . . . . . . . . . 159 6.1.5 Gaussian Process Inverse Reinforcement Learning . . . . . 164 6.1.6 Inverse Reinforcement Learning with Leveraged Gaussian Processes . . . . . . . . . . . . . . . . . . . . . . . . . . . . 167 6.1.7 Simulations and Experiment . . . . . . . . . . . . . . . . . 175 6.1.8 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . 181 7 Conclusion 185 Appendices 189 A Proofs of Chapter 3.1. 191 A.1 Useful Properties . . . . . . . . . . . . . . . . . . . . . . . . . . . . 191 A.2 Sparse Bellman Optimality Equation . . . . . . . . . . . . . . . . . 192 A.3 Sparse Tsallis Entropy . . . . . . . . . . . . . . . . . . . . . . . . . 195 A.4 Upper and Lower Bounds for Sparsemax Operation . . . . . . . . . 196 A.5 Comparison to Log-Sum-Exp . . . . . . . . . . . . . . . . . . . . . 200 A.6 Convergence and Optimality of Sparse Value Iteration . . . . . . . 201 A.7 Performance Error Bounds for Sparse Value Iteration . . . . . . . . 203 B Proofs of Chapter 3.2. 209 B.1 Change of Variables . . . . . . . . . . . . . . . . . . . . . . . . . . 209 B.2 Concavity of Maximum Causal Tsallis Entropy . . . . . . . . . . . 210 B.3 Optimality Condition of Maximum Causal Tsallis Entropy . . . . . 212 B.4 Interpretation as Robust Bayes . . . . . . . . . . . . . . . . . . . . 215 B.5 Generative Adversarial Setting with Maximum Causal Tsallis Entropy . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 216 B.6 Tsallis Entropy of a Mixture of Gaussians . . . . . . . . . . . . . . 217 B.7 Causal Entropy Approximation . . . . . . . . . . . . . . . . . . . . 218 C Proofs of Chapter 4.1. 221 C.1 q-Maximum: Bounded Approximation of Maximum . . . . . . . . . 223 C.2 Tsallis Bellman Optimality Equation . . . . . . . . . . . . . . . . . 226 C.3 Variable Change . . . . . . . . . . . . . . . . . . . . . . . . . . . . 228 C.4 Tsallis Bellman Optimality Equation . . . . . . . . . . . . . . . . . 230 C.5 Tsallis Policy Iteration . . . . . . . . . . . . . . . . . . . . . . . . . 234 C.6 Tsallis Bellman Expectation (TBE) Equation . . . . . . . . . . . . 234 C.7 Tsallis Bellman Expectation Operator and Tsallis Policy Evaluation235 C.8 Tsallis Policy Improvement . . . . . . . . . . . . . . . . . . . . . . 237 C.9 Tsallis Value Iteration . . . . . . . . . . . . . . . . . . . . . . . . . 239 C.10 Performance Error Bounds . . . . . . . . . . . . . . . . . . . . . . 241 C.11 q-Scheduling . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 243 D Proofs of Chapter 4.2. 245 D.1 In nite Exploration . . . . . . . . . . . . . . . . . . . . . . . . . . . 245 D.2 Regret Bound . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 247 E Proofs of Chapter 5.1. 255 E.1 General Regret Lower Bound of APE . . . . . . . . . . . . . . . . . 255 E.2 General Regret Upper Bound of APE . . . . . . . . . . . . . . . . 257 E.3 Proofs of Corollaries . . . . . . . . . . . . . . . . . . . . . . . . . . 266 F Proofs of Chapter 5.2. 279 F.1 Regret Lower Bound for Robust Upper Con dence Bound . . . . . 279 F.2 Bounds on Tail Probability of A p-Robust Estimator . . . . . . . . 284 F.3 General Regret Upper Bound of APE2 . . . . . . . . . . . . . . . . 287 F.4 General Regret Lower Bound of APE2 . . . . . . . . . . . . . . . . 299 F.5 Proofs of Corollaries . . . . . . . . . . . . . . . . . . . . . . . . . . 302Docto

Many-agent Reinforcement Learning

Multi-agent reinforcement learning (RL) solves the problem of how each agent should behave optimally in a stochastic environment in which multiple agents are learning simultaneously. It is an interdisciplinary domain with a long history that lies in the joint area of psychology, control theory, game theory, reinforcement learning, and deep learning. Following the remarkable success of the AlphaGO series in single-agent RL, 2019 was a booming year that witnessed significant advances in multi-agent RL techniques; impressive breakthroughs have been made on developing AIs that outperform humans on many challenging tasks, especially multi-player video games. Nonetheless, one of the key challenges of multi-agent RL techniques is the scalability; it is still non-trivial to design efficient learning algorithms that can solve tasks including far more than two agents ($N \gg 2$), which I name by \emph{many-agent reinforcement learning} (MARL\footnote{I use the world of ``MARL" to denote multi-agent reinforcement learning with a particular focus on the cases of many agents; otherwise, it is denoted as ``Multi-Agent RL" by default.}) problems. In this thesis, I contribute to tackling MARL problems from four aspects. Firstly, I offer a self-contained overview of multi-agent RL techniques from a game-theoretical perspective. This overview fills the research gap that most of the existing work either fails to cover the recent advances since 2010 or does not pay adequate attention to game theory, which I believe is the cornerstone to solving many-agent learning problems. Secondly, I develop a tractable policy evaluation algorithm -- $\alpha^\alpha$-Rank -- in many-agent systems. The critical advantage of $\alpha^\alpha$-Rank is that it can compute the solution concept of $\alpha$-Rank tractably in multi-player general-sum games with no need to store the entire pay-off matrix. This is in contrast to classic solution concepts such as Nash equilibrium which is known to be $PPAD$-hard in even two-player cases. $\alpha^\alpha$-Rank allows us, for the first time, to practically conduct large-scale multi-agent evaluations. Thirdly, I introduce a scalable policy learning algorithm -- mean-field MARL -- in many-agent systems. The mean-field MARL method takes advantage of the mean-field approximation from physics, and it is the first provably convergent algorithm that tries to break the curse of dimensionality for MARL tasks. With the proposed algorithm, I report the first result of solving the Ising model and multi-agent battle games through a MARL approach. Fourthly, I investigate the many-agent learning problem in open-ended meta-games (i.e., the game of a game in the policy space). Specifically, I focus on modelling the behavioural diversity in meta-games, and developing algorithms that guarantee to enlarge diversity during training. The proposed metric based on determinantal point processes serves as the first mathematically rigorous definition for diversity. Importantly, the diversity-aware learning algorithms beat the existing state-of-the-art game solvers in terms of exploitability by a large margin. On top of the algorithmic developments, I also contribute two real-world applications of MARL techniques. Specifically, I demonstrate the great potential of applying MARL to study the emergent population dynamics in nature, and model diverse and realistic interactions in autonomous driving. Both applications embody the prospect that MARL techniques could achieve huge impacts in the real physical world, outside of purely video games

Expert iteration

In this thesis, we study how reinforcement learning algorithms can tackle classical board games without recourse to human knowledge. Specifically, we develop a framework and algorithms which learn to play the board game Hex starting from random play. We first describe Expert Iteration (ExIt), a novel reinforcement learning framework which extends Modified Policy Iteration. ExIt explicitly decomposes the reinforcement learning problem into two parts: planning and generalisation. A planning algorithm explores possible move sequences starting from a particular position to find good strategies from that position, while a parametric function approximator is trained to predict those plans, generalising to states not yet seen. Subsequently, planning is improved by using the approximated policy to guide search, increasing the strength of new plans. This decomposition allows ExIt to combine the benefits of both planning methods and function approximation methods. We demonstrate the effectiveness of the ExIt paradigm by implementing ExIt with two different planning algorithms. First, we develop a version based on Monte Carlo Tree Search (MCTS), a search algorithm which has been successful both in specific games, such as Go, Hex and Havannah, and in general game playing competitions. We then develop a new planning algorithm, Policy Gradient Search (PGS), which uses a model-free reinforcement learning algorithm for online planning. Unlike MCTS, PGS does not require an explicit search tree. Instead PGS uses function approximation within a single search, allowing it to be applied to problems with larger branching factors. Both MCTS-ExIt and PGS-ExIt defeated MoHex 2.0 - the most recent Hex Olympiad winner to be open sourced - in 9 × 9 Hex. More importantly, whereas MoHex makes use of many Hex-specific improvements and knowledge, all our programs were trained tabula rasa using general reinforcement learning methods. This bodes well for ExIt’s applicability to both other games and real world decision making problems

ISIPTA'07: Proceedings of the Fifth International Symposium on Imprecise Probability: Theories and Applications

B