A Survey on Causal Reinforcement Learning

While Reinforcement Learning (RL) achieves tremendous success in sequential decision-making problems of many domains, it still faces key challenges of data inefficiency and the lack of interpretability. Interestingly, many researchers have leveraged insights from the causality literature recently, bringing forth flourishing works to unify the merits of causality and address well the challenges from RL. As such, it is of great necessity and significance to collate these Causal Reinforcement Learning (CRL) works, offer a review of CRL methods, and investigate the potential functionality from causality toward RL. In particular, we divide existing CRL approaches into two categories according to whether their causality-based information is given in advance or not. We further analyze each category in terms of the formalization of different models, ranging from the Markov Decision Process (MDP), Partially Observed Markov Decision Process (POMDP), Multi-Arm Bandits (MAB), and Dynamic Treatment Regime (DTR). Moreover, we summarize the evaluation matrices and open sources while we discuss emerging applications, along with promising prospects for the future development of CRL.Comment: 29 pages, 20 figure

Air quality and mental health: evidence, challenges and future directions

Background: Poor air quality is associated with poor health. Little attention is given to the complex array of environmental exposures and air pollutants that affect mental health during the life course. // Aims: We gather interdisciplinary expertise and knowledge across the air pollution and mental health fields. We seek to propose future research priorities and how to address them. // Method: Through a rapid narrative review, we summarise the key scientific findings, knowledge gaps and methodological challenges. // Results: There is emerging evidence of associations between poor air quality, both indoors and outdoors, and poor mental health more generally, as well as specific mental disorders. Furthermore, pre-existing long-term conditions appear to deteriorate, requiring more healthcare. Evidence of critical periods for exposure among children and adolescents highlights the need for more longitudinal data as the basis of early preventive actions and policies. Particulate matter, including bioaerosols, are implicated, but form part of a complex exposome influenced by geography, deprivation, socioeconomic conditions and biological and individual vulnerabilities. Critical knowledge gaps need to be addressed to design interventions for mitigation and prevention, reflecting ever-changing sources of air pollution. The evidence base can inform and motivate multi-sector and interdisciplinary efforts of researchers, practitioners, policy makers, industry, community groups and campaigners to take informed action. // Conclusions: There are knowledge gaps and a need for more research, for example, around bioaerosols exposure, indoor and outdoor pollution, urban design and impact on mental health over the life course

Broad-Scale Climate Influences on Spring-Spawning Herring (Clupea harengus, L.) Recruitment in the Western Baltic Sea

Climate forcing in complex ecosystems can have profound implications for ecosystem sustainability and may thus challenge a precautionary ecosystem management. Climatic influences documented to affect various ecological functions on a global scale, may themselves be observed on quantitative or qualitative scales including regime shifts in complex marine ecosystems. This study investigates the potential climatic impact on the reproduction success of spring-spawning herring (Clupea harengus) in the Western Baltic Sea (WBSS herring). To test for climate effects on reproduction success, the regionally determined and scientifically well-documented spawning grounds of WBSS herring represent an ideal model system. Climate effects on herring reproduction were investigated using two global indices of atmospheric variability and sea surface temperature, represented by the North Atlantic Oscillation (NAO) and the Atlantic Multi-decadal Oscillation (AMO), respectively, and the Baltic Sea Index (BSI) which is a regional-scale atmospheric index for the Baltic Sea. Moreover, we combined a traditional approach with modern time series analysis based on a recruitment model connecting parental population components with reproduction success. Generalized transfer functions (ARIMAX models) allowed evaluating the dynamic nature of exogenous climate processes interacting with the endogenous recruitment process. Using different model selection criteria our results reveal that in contrast to NAO and AMO, the BSI shows a significant positive but delayed signal on the annual dynamics of herring recruitment. The westward influence of the Siberian high is considered strongly suppressing the influence of the NAO in this area leading to a higher explanatory power of the BSI reflecting the atmospheric pressure regime on a North-South transect between Oslo, Norway and Szczecin, Poland. We suggest incorporating climate-induced effects into stock and risk assessments and management strategies as part of the EU ecosystem approach to support sustainable herring fisheries in the Western Baltic Sea

Learning how to act: making good decisions with machine learning

This thesis is about machine learning and statistical approaches to decision making. How can we learn from data to anticipate the consequence of, and optimally select, interventions or actions? Problems such as deciding which medication to prescribe to patients, who should be released on bail, and how much to charge for insurance are ubiquitous, and have far reaching impacts on our lives. There are two fundamental approaches to learning how to act: reinforcement learning, in which an agent directly intervenes in a system and learns from the outcome, and observational causal inference, whereby we seek to infer the outcome of an intervention from observing the system. The goal of this thesis to connect and unify these key approaches. I introduce causal bandit problems: a synthesis that combines causal graphical models, which were developed for observational causal inference, with multi-armed bandit problems, which are a subset of reinforcement learning problems that are simple enough to admit formal analysis. I show that knowledge of the causal structure allows us to transfer information learned about the outcome of one action to predict the outcome of an alternate action, yielding a novel form of structure between bandit arms that cannot be exploited by existing algorithms. I propose an algorithm for causal bandit problems and prove bounds on the simple regret demonstrating it is close to mini-max optimal and better than algorithms that do not use the additional causal information

Discovering Causal Relations and Equations from Data

Physics is a field of science that has traditionally used the scientific method to answer questions about why natural phenomena occur and to make testable models that explain the phenomena. Discovering equations, laws and principles that are invariant, robust and causal explanations of the world has been fundamental in physical sciences throughout the centuries. Discoveries emerge from observing the world and, when possible, performing interventional studies in the system under study. With the advent of big data and the use of data-driven methods, causal and equation discovery fields have grown and made progress in computer science, physics, statistics, philosophy, and many applied fields. All these domains are intertwined and can be used to discover causal relations, physical laws, and equations from observational data. This paper reviews the concepts, methods, and relevant works on causal and equation discovery in the broad field of Physics and outlines the most important challenges and promising future lines of research. We also provide a taxonomy for observational causal and equation discovery, point out connections, and showcase a complete set of case studies in Earth and climate sciences, fluid dynamics and mechanics, and the neurosciences. This review demonstrates that discovering fundamental laws and causal relations by observing natural phenomena is being revolutionised with the efficient exploitation of observational data, modern machine learning algorithms and the interaction with domain knowledge. Exciting times are ahead with many challenges and opportunities to improve our understanding of complex systems.Comment: 137 page