Generation of Policy-Level Explanations for Reinforcement Learning

Though reinforcement learning has greatly benefited from the incorporation of neural networks, the inability to verify the correctness of such systems limits their use. Current work in explainable deep learning focuses on explaining only a single decision in terms of input features, making it unsuitable for explaining a sequence of decisions. To address this need, we introduce Abstracted Policy Graphs, which are Markov chains of abstract states. This representation concisely summarizes a policy so that individual decisions can be explained in the context of expected future transitions. Additionally, we propose a method to generate these Abstracted Policy Graphs for deterministic policies given a learned value function and a set of observed transitions, potentially off-policy transitions used during training. Since no restrictions are placed on how the value function is generated, our method is compatible with many existing reinforcement learning methods. We prove that the worst-case time complexity of our method is quadratic in the number of features and linear in the number of provided transitions, $O(|F|^2 |tr\_samples|)$. By applying our method to a family of domains, we show that our method scales well in practice and produces Abstracted Policy Graphs which reliably capture relationships within these domains.Comment: Accepted to Proceedings of the Thirty-Third AAAI Conference on Artificial Intelligence (2019

What\u27s in a Name? The Matrix as an Introduction to Mathematics

In my classes on the nature of scientific thought, I have often used the movie The Matrix (1999) to illustrate how evidence shapes the reality we perceive (or think we perceive). As a mathematician and self-confessed science fiction fan, I usually field questions related to the movie whenever the subject of linear algebra arises, since this field is the study of matrices and their properties. So it is natural to ask, why does the movie title reference a mathematical object? Of course, there are many possible explanations for this, each of which probably contributed a little to the naming decision. First off, it sounds cool and mysterious. That much is clear, and it may be that this reason is the most heavily weighted of them all. However, a quick look at the definitions of the word reveals deeper possibilities for the meaning of the movie’s title. Consider the following definitions related to different fields of study taken from Wikipedia on January 4, 2010: • Matrix (mathematics), a mathematical object generally represented as an array of numbers. • Matrix (biology), with numerous meanings, often referring to a biological material where specialized structures are formed or embedded. • Matrix (archeology), the soil or sediment surrounding a dig site. • Matrix (geology), the fine grains between larger grains in igneous or sedimentary rocks. • Matrix (chemistry), a continuous solid phase in which particles (atoms, molecules, ions, etc.) are embedded. All of these point to an essential commonality: a matrix is an underlying structure in which other objects are embedded. This is to be expected, I suppose, given that the word is derived from the Latin word referring to the womb — something in which all of us are embedded at the beginning of our existence. And so mathematicians, being the Latin scholars we are, have adapted the term: a mathematical matrix has quantities (usually numbers, but they could be almost anything) embedded in it. A biological matrix has cell components embedded in it. A geological matrix has grains of rock embedded in it. And so on. So a second reason for the cool name is that we are talking, in the movie, about a computer system generating a virtual reality in which human beings are embedded (literally, since they are lying down in pods). Thus, the computer program forms a literal matrix, one that bears an intentional likeness to a womb. However, there are other ways to connect the idea of a matrix to the film’s premise. These explanations operate on a higher level and are explicitly relevant to the mathematical definition of a matrix as well as to the events in the trilogy of Matrix movies. They are related to computer graphics, Markov chains, and network theory. This essay will explore each of these in turn, and discuss their application to either the events in the film’s story-line or to the making of the movie itself