Survey of Recent Multi-Agent Reinforcement Learning Algorithms Utilizing Centralized Training

Much work has been dedicated to the exploration of Multi-Agent Reinforcement Learning (MARL) paradigms implementing a centralized learning with decentralized execution (CLDE) approach to achieve human-like collaboration in cooperative tasks. Here, we discuss variations of centralized training and describe a recent survey of algorithmic approaches. The goal is to explore how different implementations of information sharing mechanism in centralized learning may give rise to distinct group coordinated behaviors in multi-agent systems performing cooperative tasks.Comment: This article appeared in the news at: https://www.army.mil/article/247261/army_researchers_develop_innovative_framework_for_training_a

A Survey on Causal Reinforcement Learning

While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, many researchers have leveraged insights from the causality literature recently, bringing forth flourishing works to unify the merits of causality and address well the challenges from RL. As such, it is of great necessity and significance to collate these Causal Reinforcement Learning (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We further analyze each category in terms of the formalization of different models, ranging from the Markov Decision Process (MDP), Partially Observed Markov Decision Process (POMDP), Multi-Arm Bandits (MAB), and Dynamic Treatment Regime (DTR). Moreover, we summarize the evaluation matrices and open sources while we discuss emerging applications, along with promising prospects for the future development of CRL.Comment: 29 pages, 20 figure

Bayesian Learning in the Counterfactual World

Recent years have witnessed a surging interest towards the use of machine learning tools for causal inference. In contrast to the usual large data settings where the primary goal is prediction, many disciplines, such as health, economic and social sciences, are instead interested in causal questions. Learning individualized responses to an intervention is a crucial task in many applied fields (e.g., precision medicine, targeted advertising, precision agriculture, etc.) where the ultimate goal is to design optimal and highly-personalized policies based on individual features. In this work, I thus tackle the problem of estimating causal effects of an intervention that are heterogeneous across a population of interest and depend on an individual set of characteristics (e.g., a patient's clinical record, user's browsing history, etc..) in high-dimensional observational data settings. This is done by utilizing Bayesian Nonparametric or Probabilistic Machine Learning tools that are specifically adjusted for the causal setting and have desirable uncertainty quantification properties, with a focus on the issues of interpretability/explainability and inclusion of domain experts' prior knowledge. I begin by introducing terminology and concepts from causality and causal reasoning in the first chapter. Then I include a literature review of some of the state-of-the-art regression-based methods for heterogeneous treatment effects estimation, with an attempt to build a unifying taxonomy and lay down the finite-sample empirical properties of these models. The chapters forming the core of the dissertation instead present some novel methods addressing existing issues in individualized causal effects estimation: Chapter 3 develops both a Bayesian tree ensemble method and a deep learning architecture to tackle interpretability, uncertainty coverage and targeted regularization; Chapter 4 instead introduces a novel multi-task Deep Kernel Learning method particularly suited for multi-outcome | multi-action scenarios. The last chapter concludes with a discussion

Some considerations on the estimation of the value associated to a clinical act

The assignment of a value to any economic system, especially in healthcare management, is the topic of this article. The assignment of a value to a clinical act is a very complex process, as it joins the complexity of estimating value in an economic system with the estimation of the value of well-being. An interdisciplinary approach joining disciplines such as Philosophy, Business, Psychology and Physics is used to analyse the assignment of a value; and it is obtained that it is necessary the integrated use of three concepts; viz., Truth, Good, and Beauty. It is also obtained that the concept of Beauty has the biggest difficulty in being computationally represented, and that to achieve such representation it is necessary the use of Statistical Philosophy, a here-proposed branch of the Philosophy of Information. Moreover, it is obtained that value is made of three types of value; viz., Truth-value, Good-value, and Beauty-value. Finally, it is made an assessment of the difficulty in choosing the appropriate necessary projection of the 3-vector value into a worthiness-scalar, a projection that is necessary because the choice of a best option, e.g. a best clinical act, always requires that the option is quantified by a scalar. (C) 2020 The Authors. Published by Elsevier B.V.NFL thanks Eduarda Sousa for support. Thanks to Sandra Lori for the drawings. All the funding was provided by FCT (Fundacao para a Ciencia e a Tecnologia): NFL was funded by a fellowship of project MEDPERSYST-POCI01-0145-FEDER-016428 and by the INESC-ID multiannual funding from the PIDDAC program (UID/CEC/50021/2020); and the work of both JN and VA has been supported within the project scope of UID/CEC/00319/2020