DeepSynth: Automata Synthesis for Automatic Task Segmentation in Deep Reinforcement Learning

This paper proposes DeepSynth, a method for effective training of deep Reinforcement Learning (RL) agents when the reward is sparse and non-Markovian, but at the same time progress towards the reward requires achieving an unknown sequence of high-level objectives. Our method employs a novel algorithm for synthesis of compact automata to uncover this sequential structure automatically. We synthesise a human-interpretable automaton from trace data collected by exploring the environment. The state space of the environment is then enriched with the synthesised automaton so that the generation of a control policy by deep RL is guided by the discovered structure encoded in the automaton. The proposed approach is able to cope with both high-dimensional, low-level features and unknown sparse non-Markovian rewards. We have evaluated DeepSynth's performance in a set of experiments that includes the Atari game Montezuma's Revenge. Compared to existing approaches, we obtain a reduction of two orders of magnitude in the number of iterations required for policy synthesis, and also a significant improvement in scalability.Comment: Extended version of AAAI 2021 pape

Reasoning about causality in games

Causal reasoning and game-theoretic reasoning are fundamental topics in artificial intelligence, among many other disciplines: this paper is concerned with their intersection. Despite their importance, a formal framework that supports both these forms of reasoning has, until now, been lacking. We offer a solution in the form of (structural) causal games, which can be seen as extending Pearl's causal hierarchy to the game-theoretic domain, or as extending Koller and Milch's multi-agent influence diagrams to the causal domain. We then consider three key questions: i) How can the (causal) dependencies in games – either between variables, or between strategies – be modelled in a uniform, principled manner? ii) How may causal queries be computed in causal games, and what assumptions does this require? iii) How do causal games compare to existing formalisms? To address question i), we introduce mechanised games, which encode dependencies between agents' decision rules and the distributions governing the game. In response to question ii), we present definitions of predictions, interventions, and counterfactuals, and discuss the assumptions required for each. Regarding question iii), we describe correspondences between causal games and other formalisms, and explain how causal games can be used to answer queries that other causal or game-theoretic models do not support. Finally, we highlight possible applications of causal games, aided by an extensive open-source Python library

Infrared Nanoimaging of Hydrogenated Perovskite Nickelate Synaptic Devices

Solid-state devices made from correlated oxides such as perovskite nickelates are promising for neuromorphic computing by mimicking biological synaptic function. However, comprehending dopant action at the nanoscale poses a formidable challenge to understanding the elementary mechanisms involved. Here, we perform operando infrared nanoimaging of hydrogen-doped correlated perovskite, neodymium nickel oxide (H-NdNiO3) devices and reveal how an applied field perturbs dopant distribution at the nanoscale. This perturbation leads to stripe phases of varying conductivity perpendicular to the applied field, which define the macroscale electrical characteristics of the devices. Hyperspectral nano-FTIR imaging in conjunction with density functional theory calculations unveil a real-space map of multiple vibrational states of H-NNO associated with OH stretching modes and their dependence on the dopant concentration. Moreover, the localization of excess charges induces an out-of-plane lattice expansion in NNO which was confirmed by in-situ - x-ray diffraction and creates a strain that acts as a barrier against further diffusion. Our results and the techniques presented here hold great potential to the rapidly growing field of memristors and neuromorphic devices wherein nanoscale ion motion is fundamentally responsible for function.Comment: 30 pages, 5 figures in the main text and 5 figures in the Supplementary Materia

Global, regional, and national burden of Alzheimer's disease and other dementias, 1990-2016: a systematic analysis for the Global Burden of Disease Study 2016.

BACKGROUND: The number of individuals living with dementia is increasing, negatively affecting families, communities, and health-care systems around the world. A successful response to these challenges requires an accurate understanding of the dementia disease burden. We aimed to present the first detailed analysis of the global prevalence, mortality, and overall burden of dementia as captured by the Global Burden of Diseases, Injuries, and Risk Factors (GBD) Study 2016, and highlight the most important messages for clinicians and neurologists. METHODS: GBD 2016 obtained data on dementia from vital registration systems, published scientific literature and surveys, and data from health-service encounters on deaths, excess mortality, prevalence, and incidence from 195 countries and territories from 1990 to 2016, through systematic review and additional data-seeking efforts. To correct for differences in cause of death coding across time and locations, we modelled mortality due to dementia using prevalence data and estimates of excess mortality derived from countries that were most likely to code deaths to dementia relative to prevalence. Data were analysed by standardised methods to estimate deaths, prevalence, years of life lost (YLLs), years of life lived with disability (YLDs), and disability-adjusted life-years (DALYs; computed as the sum of YLLs and YLDs), and the fractions of these metrics that were attributable to four risk factors that met GBD criteria for assessment (high body-mass index [BMI], high fasting plasma glucose, smoking, and a diet high in sugar-sweetened beverages). FINDINGS: In 2016, the global number of individuals who lived with dementia was 43·8 million (95% uncertainty interval [UI] 37·8-51·0), increased from 20.2 million (17·4-23·5) in 1990. This increase of 117% (95% UI 114-121) contrasted with a minor increase in age-standardised prevalence of 1·7% (1·0-2·4), from 701 cases (95% UI 602-815) per 100 000 population in 1990 to 712 cases (614-828) per 100 000 population in 2016. More women than men had dementia in 2016 (27·0 million, 95% UI 23·3-31·4, vs 16.8 million, 14.4-19.6), and dementia was the fifth leading cause of death globally, accounting for 2·4 million (95% UI 2·1-2·8) deaths. Overall, 28·8 million (95% UI 24·5-34·0) DALYs were attributed to dementia; 6·4 million (95% UI 3·4-10·5) of these could be attributed to the modifiable GBD risk factors of high BMI, high fasting plasma glucose, smoking, and a high intake of sugar-sweetened beverages. INTERPRETATION: The global number of people living with dementia more than doubled from 1990 to 2016, mainly due to increases in population ageing and growth. Although differences in coding for causes of death and the heterogeneity in case-ascertainment methods constitute major challenges to the estimation of the burden of dementia, future analyses should improve on the methods for the correction of these biases. Until breakthroughs are made in prevention or curative treatment, dementia will constitute an increasing challenge to health-care systems worldwide

The impact of surgical delay on resectability of colorectal cancer: An international prospective cohort study

AIM: The SARS-CoV-2 pandemic has provided a unique opportunity to explore the impact of surgical delays on cancer resectability. This study aimed to compare resectability for colorectal cancer patients undergoing delayed versus non-delayed surgery. METHODS: This was an international prospective cohort study of consecutive colorectal cancer patients with a decision for curative surgery (January-April 2020). Surgical delay was defined as an operation taking place more than 4 weeks after treatment decision, in a patient who did not receive neoadjuvant therapy. A subgroup analysis explored the effects of delay in elective patients only. The impact of longer delays was explored in a sensitivity analysis. The primary outcome was complete resection, defined as curative resection with an R0 margin. RESULTS: Overall, 5453 patients from 304 hospitals in 47 countries were included, of whom 6.6% (358/5453) did not receive their planned operation. Of the 4304 operated patients without neoadjuvant therapy, 40.5% (1744/4304) were delayed beyond 4 weeks. Delayed patients were more likely to be older, men, more comorbid, have higher body mass index and have rectal cancer and early stage disease. Delayed patients had higher unadjusted rates of complete resection (93.7% vs. 91.9%, P = 0.032) and lower rates of emergency surgery (4.5% vs. 22.5%, P < 0.001). After adjustment, delay was not associated with a lower rate of complete resection (OR 1.18, 95% CI 0.90-1.55, P = 0.224), which was consistent in elective patients only (OR 0.94, 95% CI 0.69-1.27, P = 0.672). Longer delays were not associated with poorer outcomes. CONCLUSION: One in 15 colorectal cancer patients did not receive their planned operation during the first wave of COVID-19. Surgical delay did not appear to compromise resectability, raising the hypothesis that any reduction in long-term survival attributable to delays is likely to be due to micro-metastatic disease

Mortality and pulmonary complications in patients undergoing surgery with perioperative SARS-CoV-2 infection: an international cohort study

Background: The impact of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) on postoperative recovery needs to be understood to inform clinical decision making during and after the COVID-19 pandemic. This study reports 30-day mortality and pulmonary complication rates in patients with perioperative SARS-CoV-2 infection. Methods: This international, multicentre, cohort study at 235 hospitals in 24 countries included all patients undergoing surgery who had SARS-CoV-2 infection confirmed within 7 days before or 30 days after surgery. The primary outcome measure was 30-day postoperative mortality and was assessed in all enrolled patients. The main secondary outcome measure was pulmonary complications, defined as pneumonia, acute respiratory distress syndrome, or unexpected postoperative ventilation. Findings: This analysis includes 1128 patients who had surgery between Jan 1 and March 31, 2020, of whom 835 (74·0%) had emergency surgery and 280 (24·8%) had elective surgery. SARS-CoV-2 infection was confirmed preoperatively in 294 (26·1%) patients. 30-day mortality was 23·8% (268 of 1128). Pulmonary complications occurred in 577 (51·2%) of 1128 patients; 30-day mortality in these patients was 38·0% (219 of 577), accounting for 81·7% (219 of 268) of all deaths. In adjusted analyses, 30-day mortality was associated with male sex (odds ratio 1·75 [95% CI 1·28–2·40], p\textless0·0001), age 70 years or older versus younger than 70 years (2·30 [1·65–3·22], p\textless0·0001), American Society of Anesthesiologists grades 3–5 versus grades 1–2 (2·35 [1·57–3·53], p\textless0·0001), malignant versus benign or obstetric diagnosis (1·55 [1·01–2·39], p=0·046), emergency versus elective surgery (1·67 [1·06–2·63], p=0·026), and major versus minor surgery (1·52 [1·01–2·31], p=0·047). Interpretation: Postoperative pulmonary complications occur in half of patients with perioperative SARS-CoV-2 infection and are associated with high mortality. Thresholds for surgery during the COVID-19 pandemic should be higher than during normal practice, particularly in men aged 70 years and older. Consideration should be given for postponing non-urgent procedures and promoting non-operative treatment to delay or avoid the need for surgery. Funding: National Institute for Health Research (NIHR), Association of Coloproctology of Great Britain and Ireland, Bowel and Cancer Research, Bowel Disease Research Foundation, Association of Upper Gastrointestinal Surgeons, British Association of Surgical Oncology, British Gynaecological Cancer Society, European Society of Coloproctology, NIHR Academy, Sarcoma UK, Vascular Society for Great Britain and Ireland, and Yorkshire Cancer Research

Global, regional, and national burden of disorders affecting the nervous system, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021

BACKGROUND: Disorders affecting the nervous system are diverse and include neurodevelopmental disorders, late-life neurodegeneration, and newly emergent conditions, such as cognitive impairment following COVID-19. Previous publications from the Global Burden of Disease, Injuries, and Risk Factor Study estimated the burden of 15 neurological conditions in 2015 and 2016, but these analyses did not include neurodevelopmental disorders, as defined by the International Classification of Diseases (ICD)-11, or a subset of cases of congenital, neonatal, and infectious conditions that cause neurological damage. Here, we estimate nervous system health loss caused by 37 unique conditions and their associated risk factors globally, regionally, and nationally from 1990 to 2021. METHODS: We estimated mortality, prevalence, years lived with disability (YLDs), years of life lost (YLLs), and disability-adjusted life-years (DALYs), with corresponding 95% uncertainty intervals (UIs), by age and sex in 204 countries and territories, from 1990 to 2021. We included morbidity and deaths due to neurological conditions, for which health loss is directly due to damage to the CNS or peripheral nervous system. We also isolated neurological health loss from conditions for which nervous system morbidity is a consequence, but not the primary feature, including a subset of congenital conditions (ie, chromosomal anomalies and congenital birth defects), neonatal conditions (ie, jaundice, preterm birth, and sepsis), infectious diseases (ie, COVID-19, cystic echinococcosis, malaria, syphilis, and Zika virus disease), and diabetic neuropathy. By conducting a sequela-level analysis of the health outcomes for these conditions, only cases where nervous system damage occurred were included, and YLDs were recalculated to isolate the non-fatal burden directly attributable to nervous system health loss. A comorbidity correction was used to calculate total prevalence of all conditions that affect the nervous system combined. FINDINGS: Globally, the 37 conditions affecting the nervous system were collectively ranked as the leading group cause of DALYs in 2021 (443 million, 95% UI 378–521), affecting 3·40 billion (3·20–3·62) individuals (43·1%, 40·5–45·9 of the global population); global DALY counts attributed to these conditions increased by 18·2% (8·7–26·7) between 1990 and 2021. Age-standardised rates of deaths per 100 000 people attributed to these conditions decreased from 1990 to 2021 by 33·6% (27·6–38·8), and age-standardised rates of DALYs attributed to these conditions decreased by 27·0% (21·5–32·4). Age-standardised prevalence was almost stable, with a change of 1·5% (0·7–2·4). The ten conditions with the highest age-standardised DALYs in 2021 were stroke, neonatal encephalopathy, migraine, Alzheimer's disease and other dementias, diabetic neuropathy, meningitis, epilepsy, neurological complications due to preterm birth, autism spectrum disorder, and nervous system cancer. INTERPRETATION: As the leading cause of overall disease burden in the world, with increasing global DALY counts, effective prevention, treatment, and rehabilitation strategies for disorders affecting the nervous system are needed

Global, regional, and national disability-adjusted life-years (DALYs) for 315 diseases and injuries and healthy life expectancy (HALE), 1990-2015: a systematic analysis for the Global Burden of Disease Study 2015.

BACKGROUND: Healthy life expectancy (HALE) and disability-adjusted life-years (DALYs) provide summary measures of health across geographies and time that can inform assessments of epidemiological patterns and health system performance, help to prioritise investments in research and development, and monitor progress toward the Sustainable Development Goals (SDGs). We aimed to provide updated HALE and DALYs for geographies worldwide and evaluate how disease burden changes with development. METHODS: We used results from the Global Burden of Diseases, Injuries, and Risk Factors Study 2015 (GBD 2015) for all-cause mortality, cause-specific mortality, and non-fatal disease burden to derive HALE and DALYs by sex for 195 countries and territories from 1990 to 2015. We calculated DALYs by summing years of life lost (YLLs) and years of life lived with disability (YLDs) for each geography, age group, sex, and year. We estimated HALE using the Sullivan method, which draws from age-specific death rates and YLDs per capita. We then assessed how observed levels of DALYs and HALE differed from expected trends calculated with the Socio-demographic Index (SDI), a composite indicator constructed from measures of income per capita, average years of schooling, and total fertility rate. FINDINGS: Total global DALYs remained largely unchanged from 1990 to 2015, with decreases in communicable, neonatal, maternal, and nutritional (Group 1) disease DALYs offset by increased DALYs due to non-communicable diseases (NCDs). Much of this epidemiological transition was caused by changes in population growth and ageing, but it was accelerated by widespread improvements in SDI that also correlated strongly with the increasing importance of NCDs. Both total DALYs and age-standardised DALY rates due to most Group 1 causes significantly decreased by 2015, and although total burden climbed for the majority of NCDs, age-standardised DALY rates due to NCDs declined. Nonetheless, age-standardised DALY rates due to several high-burden NCDs (including osteoarthritis, drug use disorders, depression, diabetes, congenital birth defects, and skin, oral, and sense organ diseases) either increased or remained unchanged, leading to increases in their relative ranking in many geographies. From 2005 to 2015, HALE at birth increased by an average of 2·9 years (95% uncertainty interval 2·9-3·0) for men and 3·5 years (3·4-3·7) for women, while HALE at age 65 years improved by 0·85 years (0·78-0·92) and 1·2 years (1·1-1·3), respectively. Rising SDI was associated with consistently higher HALE and a somewhat smaller proportion of life spent with functional health loss; however, rising SDI was related to increases in total disability. Many countries and territories in central America and eastern sub-Saharan Africa had increasingly lower rates of disease burden than expected given their SDI. At the same time, a subset of geographies recorded a growing gap between observed and expected levels of DALYs, a trend driven mainly by rising burden due to war, interpersonal violence, and various NCDs. INTERPRETATION: Health is improving globally, but this means more populations are spending more time with functional health loss, an absolute expansion of morbidity. The proportion of life spent in ill health decreases somewhat with increasing SDI, a relative compression of morbidity, which supports continued efforts to elevate personal income, improve education, and limit fertility. Our analysis of DALYs and HALE and their relationship to SDI represents a robust framework on which to benchmark geography-specific health performance and SDG progress. Country-specific drivers of disease burden, particularly for causes with higher-than-expected DALYs, should inform financial and research investments, prevention efforts, health policies, and health system improvement initiatives for all countries along the development continuum. FUNDING: Bill & Melinda Gates Foundation