Zero-Shot Learning with Common Sense Knowledge Graphs

Zero-shot learning relies on semantic class representations such as hand-engineered attributes or learned embeddings to predict classes without any labeled examples. We propose to learn class representations from common sense knowledge graphs. Common sense knowledge graphs are an untapped source of explicit high-level knowledge that requires little human effort to apply to a range of tasks. To capture the knowledge in the graph, we introduce ZSL-KG, a general-purpose framework with a novel transformer graph convolutional network (TrGCN) for generating class representations. Our proposed TrGCN architecture computes non-linear combinations of the node neighbourhood and shows improvements on zero-shot learning tasks in language and vision. Our results show ZSL-KG outperforms the best performing graph-based zero-shot learning framework by an average of 2.1 accuracy points with improvements as high as 3.4 accuracy points. Our ablation study on ZSL-KG with alternate graph neural networks shows that our TrGCN adds up to 1.2 accuracy points improvement on these tasks

Does CLIP Bind Concepts? Probing Compositionality in Large Image Models

Large-scale neural network models combining text and images have made incredible progress in recent years. However, it remains an open question to what extent such models encode compositional representations of the concepts over which they operate, such as correctly identifying ''red cube'' by reasoning over the constituents ''red'' and ''cube''. In this work, we focus on the ability of a large pretrained vision and language model (CLIP) to encode compositional concepts and to bind variables in a structure-sensitive way (e.g., differentiating ''cube behind sphere'' from ''sphere behind cube''). In order to inspect the performance of CLIP, we compare several architectures from research on compositional distributional semantics models (CDSMs), a line of research that attempts to implement traditional compositional linguistic structures within embedding spaces. We find that CLIP can compose concepts in a single-object setting, but in situations where concept binding is needed, performance drops dramatically. At the same time, CDSMs also perform poorly, with best performance at chance level

PromptSource: An Integrated Development Environment and Repository for Natural Language Prompts

PromptSource is a system for creating, sharing, and using natural language prompts. Prompts are functions that map an example from a dataset to a natural language input and target output. Using prompts to train and query language models is an emerging area in NLP that requires new tools that let users develop and refine these prompts collaboratively. PromptSource addresses the emergent challenges in this new setting with (1) a templating language for defining data-linked prompts, (2) an interface that lets users quickly iterate on prompt development by observing outputs of their prompts on many examples, and (3) a community-driven set of guidelines for contributing new prompts to a common pool. Over 2,000 prompts for roughly 170 datasets are already available in PromptSource. PromptSource is available at https://github.com/bigscience-workshop/promptsource.Comment: ACL 2022 Dem

Multitask Prompted Training Enables Zero-Shot Task Generalization

International audienceLarge language models have recently been shown to attain reasonable zero-shot generalization on a diverse set of tasks (Brown et al., 2020). It has been hypothesized that this is a consequence of implicit multitask learning in language models’ pretraining (Radford et al., 2019). Can zero-shot generalization instead be directly induced by explicit multitask learning? To test this question at scale, we develop a system for easily mapping any natural language tasks into a human-readable prompted form. We convert a large set of supervised datasets, each with multiple prompts with diverse wording. These prompted datasets allow for benchmarking the ability of a model to perform completely held-out tasks. We fine-tune a pre-trained encoder-decoder model (Raffel et al., 2020; Lester et al., 2021) on this multitask mixture covering a wide variety of tasks. The model attains strong zero-shot performance on several standard datasets, often outperforming models up to 16x its size. Further, our approach attains strong performance on a subset of tasks from the BIG-bench benchmark, outperforming models up to 6x its size. All trained models are available at https://github.com/bigscience-workshop/t-zero, and all prompts are available at https://github.com/bigscience-workshop/promptsource

BLOOM: A 176B-Parameter Open-Access Multilingual Language Model

Large language models (LLMs) have been shown to be able to perform new tasks based on a few demonstrations or natural language instructions. While these capabilities have led to widespread adoption, most LLMs are developed by resource-rich organizations and are frequently kept from the public. As a step towards democratizing this powerful technology, we present BLOOM, a 176B-parameter open-access language model designed and built thanks to a collaboration of hundreds of researchers. BLOOM is a decoder-only Transformer language model that was trained on the ROOTS corpus, a dataset comprising hundreds of sources in 46 natural and 13 programming languages (59 in total). We find that BLOOM achieves competitive performance on a wide variety of benchmarks, with stronger results after undergoing multitask prompted finetuning. To facilitate future research and applications using LLMs, we publicly release our models and code under the Responsible AI License

The global burden of adolescent and young adult cancer in 2019 : a systematic analysis for the Global Burden of Disease Study 2019

Background In estimating the global burden of cancer, adolescents and young adults with cancer are often overlooked, despite being a distinct subgroup with unique epidemiology, clinical care needs, and societal impact. Comprehensive estimates of the global cancer burden in adolescents and young adults (aged 15-39 years) are lacking. To address this gap, we analysed results from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2019, with a focus on the outcome of disability-adjusted life-years (DALYs), to inform global cancer control measures in adolescents and young adults. Methods Using the GBD 2019 methodology, international mortality data were collected from vital registration systems, verbal autopsies, and population-based cancer registry inputs modelled with mortality-to-incidence ratios (MIRs). Incidence was computed with mortality estimates and corresponding MIRs. Prevalence estimates were calculated using modelled survival and multiplied by disability weights to obtain years lived with disability (YLDs). Years of life lost (YLLs) were calculated as age-specific cancer deaths multiplied by the standard life expectancy at the age of death. The main outcome was DALYs (the sum of YLLs and YLDs). Estimates were presented globally and by Socio-demographic Index (SDI) quintiles (countries ranked and divided into five equal SDI groups), and all estimates were presented with corresponding 95% uncertainty intervals (UIs). For this analysis, we used the age range of 15-39 years to define adolescents and young adults. Findings There were 1.19 million (95% UI 1.11-1.28) incident cancer cases and 396 000 (370 000-425 000) deaths due to cancer among people aged 15-39 years worldwide in 2019. The highest age-standardised incidence rates occurred in high SDI (59.6 [54.5-65.7] per 100 000 person-years) and high-middle SDI countries (53.2 [48.8-57.9] per 100 000 person-years), while the highest age-standardised mortality rates were in low-middle SDI (14.2 [12.9-15.6] per 100 000 person-years) and middle SDI (13.6 [12.6-14.8] per 100 000 person-years) countries. In 2019, adolescent and young adult cancers contributed 23.5 million (21.9-25.2) DALYs to the global burden of disease, of which 2.7% (1.9-3.6) came from YLDs and 97.3% (96.4-98.1) from YLLs. Cancer was the fourth leading cause of death and tenth leading cause of DALYs in adolescents and young adults globally. Interpretation Adolescent and young adult cancers contributed substantially to the overall adolescent and young adult disease burden globally in 2019. These results provide new insights into the distribution and magnitude of the adolescent and young adult cancer burden around the world. With notable differences observed across SDI settings, these estimates can inform global and country-level cancer control efforts. Copyright (C) 2021 The Author(s). Published by Elsevier Ltd.Peer reviewe

Alcohol use and burden for 195 countries and territories, 1990-2016 : a systematic analysis for the Global Burden of Disease Study 2016

Background Alcohol use is a leading risk factor for death and disability, but its overall association with health remains complex given the possible protective effects of moderate alcohol consumption on some conditions. With our comprehensive approach to health accounting within the Global Burden of Diseases, Injuries, and Risk Factors Study 2016, we generated improved estimates of alcohol use and alcohol-attributable deaths and disability-adjusted life-years (DALYs) for 195 locations from 1990 to 2016, for both sexes and for 5-year age groups between the ages of 15 years and 95 years and older. Methods Using 694 data sources of individual and population-level alcohol consumption, along with 592 prospective and retrospective studies on the risk of alcohol use, we produced estimates of the prevalence of current drinking, abstention, the distribution of alcohol consumption among current drinkers in standard drinks daily (defined as 10 g of pure ethyl alcohol), and alcohol-attributable deaths and DALYs. We made several methodological improvements compared with previous estimates: first, we adjusted alcohol sales estimates to take into account tourist and unrecorded consumption; second, we did a new meta-analysis of relative risks for 23 health outcomes associated with alcohol use; and third, we developed a new method to quantify the level of alcohol consumption that minimises the overall risk to individual health. Findings Globally, alcohol use was the seventh leading risk factor for both deaths and DALYs in 2016, accounting for 2.2% (95% uncertainty interval [UI] 1.5-3.0) of age-standardised female deaths and 6.8% (5.8-8.0) of age-standardised male deaths. Among the population aged 15-49 years, alcohol use was the leading risk factor globally in 2016, with 3.8% (95% UI 3.2-4-3) of female deaths and 12.2% (10.8-13-6) of male deaths attributable to alcohol use. For the population aged 15-49 years, female attributable DALYs were 2.3% (95% UI 2.0-2.6) and male attributable DALYs were 8.9% (7.8-9.9). The three leading causes of attributable deaths in this age group were tuberculosis (1.4% [95% UI 1. 0-1. 7] of total deaths), road injuries (1.2% [0.7-1.9]), and self-harm (1.1% [0.6-1.5]). For populations aged 50 years and older, cancers accounted for a large proportion of total alcohol-attributable deaths in 2016, constituting 27.1% (95% UI 21.2-33.3) of total alcohol-attributable female deaths and 18.9% (15.3-22.6) of male deaths. The level of alcohol consumption that minimised harm across health outcomes was zero (95% UI 0.0-0.8) standard drinks per week. Interpretation Alcohol use is a leading risk factor for global disease burden and causes substantial health loss. We found that the risk of all-cause mortality, and of cancers specifically, rises with increasing levels of consumption, and the level of consumption that minimises health loss is zero. These results suggest that alcohol control policies might need to be revised worldwide, refocusing on efforts to lower overall population-level consumption.Peer reviewe

Adolescent transport and unintentional injuries: a systematic analysis using the Global Burden of Disease Study 2019

Background: Globally, transport and unintentional injuries persist as leading preventable causes of mortality and morbidity for adolescents. We sought to report comprehensive trends in injury-related mortality and morbidity for adolescents aged 10–24 years during the past three decades. Methods: Using the Global Burden of Disease, Injuries, and Risk Factors 2019 Study, we analysed mortality and disability-adjusted life-years (DALYs) attributed to transport and unintentional injuries for adolescents in 204 countries. Burden is reported in absolute numbers and age-standardised rates per 100 000 population by sex, age group (10–14, 15–19, and 20–24 years), and sociodemographic index (SDI) with 95% uncertainty intervals (UIs). We report percentage changes in deaths and DALYs between 1990 and 2019. Findings: In 2019, 369 061 deaths (of which 214 337 [58%] were transport related) and 31·1 million DALYs (of which 16·2 million [52%] were transport related) among adolescents aged 10–24 years were caused by transport and unintentional injuries combined. If compared with other causes, transport and unintentional injuries combined accounted for 25% of deaths and 14% of DALYs in 2019, and showed little improvement from 1990 when such injuries accounted for 26% of adolescent deaths and 17% of adolescent DALYs. Throughout adolescence, transport and unintentional injury fatality rates increased by age group. The unintentional injury burden was higher among males than females for all injury types, except for injuries related to fire, heat, and hot substances, or to adverse effects of medical treatment. From 1990 to 2019, global mortality rates declined by 34·4% (from 17·5 to 11·5 per 100 000) for transport injuries, and by 47·7% (from 15·9 to 8·3 per 100 000) for unintentional injuries. However, in low-SDI nations the absolute number of deaths increased (by 80·5% to 42 774 for transport injuries and by 39·4% to 31 961 for unintentional injuries). In the high-SDI quintile in 2010–19, the rate per 100 000 of transport injury DALYs was reduced by 16·7%, from 838 in 2010 to 699 in 2019. This was a substantially slower pace of reduction compared with the 48·5% reduction between 1990 and 2010, from 1626 per 100 000 in 1990 to 838 per 100 000 in 2010. Between 2010 and 2019, the rate of unintentional injury DALYs per 100 000 also remained largely unchanged in high-SDI countries (555 in 2010 vs 554 in 2019; 0·2% reduction). The number and rate of adolescent deaths and DALYs owing to environmental heat and cold exposure increased for the high-SDI quintile during 2010–19. Interpretation: As other causes of mortality are addressed, inadequate progress in reducing transport and unintentional injury mortality as a proportion of adolescent deaths becomes apparent. The relative shift in the burden of injury from high-SDI countries to low and low–middle-SDI countries necessitates focused action, including global donor, government, and industry investment in injury prevention. The persisting burden of DALYs related to transport and unintentional injuries indicates a need to prioritise innovative measures for the primary prevention of adolescent injury. Funding: Bill & Melinda Gates Foundation

Semaglutide and cardiovascular outcomes in patients with obesity and prevalent heart failure: a prespecified analysis of the SELECT trial

Background: Semaglutide, a GLP-1 receptor agonist, reduces the risk of major adverse cardiovascular events (MACE) in people with overweight or obesity, but the effects of this drug on outcomes in patients with atherosclerotic cardiovascular disease and heart failure are unknown. We report a prespecified analysis of the effect of once-weekly subcutaneous semaglutide 2·4 mg on ischaemic and heart failure cardiovascular outcomes. We aimed to investigate if semaglutide was beneficial in patients with atherosclerotic cardiovascular disease with a history of heart failure compared with placebo; if there was a difference in outcome in patients designated as having heart failure with preserved ejection fraction compared with heart failure with reduced ejection fraction; and if the efficacy and safety of semaglutide in patients with heart failure was related to baseline characteristics or subtype of heart failure. Methods: The SELECT trial was a randomised, double-blind, multicentre, placebo-controlled, event-driven phase 3 trial in 41 countries. Adults aged 45 years and older, with a BMI of 27 kg/m2 or greater and established cardiovascular disease were eligible for the study. Patients were randomly assigned (1:1) with a block size of four using an interactive web response system in a double-blind manner to escalating doses of once-weekly subcutaneous semaglutide over 16 weeks to a target dose of 2·4 mg, or placebo. In a prespecified analysis, we examined the effect of semaglutide compared with placebo in patients with and without a history of heart failure at enrolment, subclassified as heart failure with preserved ejection fraction, heart failure with reduced ejection fraction, or unclassified heart failure. Endpoints comprised MACE (a composite of non-fatal myocardial infarction, non-fatal stroke, and cardiovascular death); a composite heart failure outcome (cardiovascular death or hospitalisation or urgent hospital visit for heart failure); cardiovascular death; and all-cause death. The study is registered with ClinicalTrials.gov, NCT03574597. Findings: Between Oct 31, 2018, and March 31, 2021, 17 604 patients with a mean age of 61·6 years (SD 8·9) and a mean BMI of 33·4 kg/m2 (5·0) were randomly assigned to receive semaglutide (8803 [50·0%] patients) or placebo (8801 [50·0%] patients). 4286 (24·3%) of 17 604 patients had a history of investigator-defined heart failure at enrolment: 2273 (53·0%) of 4286 patients had heart failure with preserved ejection fraction, 1347 (31·4%) had heart failure with reduced ejection fraction, and 666 (15·5%) had unclassified heart failure. Baseline characteristics were similar between patients with and without heart failure. Patients with heart failure had a higher incidence of clinical events. Semaglutide improved all outcome measures in patients with heart failure at random assignment compared with those without heart failure (hazard ratio [HR] 0·72, 95% CI 0·60-0·87 for MACE; 0·79, 0·64-0·98 for the heart failure composite endpoint; 0·76, 0·59-0·97 for cardiovascular death; and 0·81, 0·66-1·00 for all-cause death; all pinteraction>0·19). Treatment with semaglutide resulted in improved outcomes in both the heart failure with reduced ejection fraction (HR 0·65, 95% CI 0·49-0·87 for MACE; 0·79, 0·58-1·08 for the composite heart failure endpoint) and heart failure with preserved ejection fraction groups (0·69, 0·51-0·91 for MACE; 0·75, 0·52-1·07 for the composite heart failure endpoint), although patients with heart failure with reduced ejection fraction had higher absolute event rates than those with heart failure with preserved ejection fraction. For MACE and the heart failure composite, there were no significant differences in benefits across baseline age, sex, BMI, New York Heart Association status, and diuretic use. Serious adverse events were less frequent with semaglutide versus placebo, regardless of heart failure subtype. Interpretation: In patients with atherosclerotic cardiovascular diease and overweight or obesity, treatment with semaglutide 2·4 mg reduced MACE and composite heart failure endpoints compared with placebo in those with and without clinical heart failure, regardless of heart failure subtype. Our findings could facilitate prescribing and result in improved clinical outcomes for this patient group. Funding: Novo Nordisk

Burden of disease scenarios for 204 countries and territories, 2022–2050: a forecasting analysis for the Global Burden of Disease Study 2021

Background: Future trends in disease burden and drivers of health are of great interest to policy makers and the public at large. This information can be used for policy and long-term health investment, planning, and prioritisation. We have expanded and improved upon previous forecasts produced as part of the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) and provide a reference forecast (the most likely future), and alternative scenarios assessing disease burden trajectories if selected sets of risk factors were eliminated from current levels by 2050. Methods: Using forecasts of major drivers of health such as the Socio-demographic Index (SDI; a composite measure of lag-distributed income per capita, mean years of education, and total fertility under 25 years of age) and the full set of risk factor exposures captured by GBD, we provide cause-specific forecasts of mortality, years of life lost (YLLs), years lived with disability (YLDs), and disability-adjusted life-years (DALYs) by age and sex from 2022 to 2050 for 204 countries and territories, 21 GBD regions, seven super-regions, and the world. All analyses were done at the cause-specific level so that only risk factors deemed causal by the GBD comparative risk assessment influenced future trajectories of mortality for each disease. Cause-specific mortality was modelled using mixed-effects models with SDI and time as the main covariates, and the combined impact of causal risk factors as an offset in the model. At the all-cause mortality level, we captured unexplained variation by modelling residuals with an autoregressive integrated moving average model with drift attenuation. These all-cause forecasts constrained the cause-specific forecasts at successively deeper levels of the GBD cause hierarchy using cascading mortality models, thus ensuring a robust estimate of cause-specific mortality. For non-fatal measures (eg, low back pain), incidence and prevalence were forecasted from mixed-effects models with SDI as the main covariate, and YLDs were computed from the resulting prevalence forecasts and average disability weights from GBD. Alternative future scenarios were constructed by replacing appropriate reference trajectories for risk factors with hypothetical trajectories of gradual elimination of risk factor exposure from current levels to 2050. The scenarios were constructed from various sets of risk factors: environmental risks (Safer Environment scenario), risks associated with communicable, maternal, neonatal, and nutritional diseases (CMNNs; Improved Childhood Nutrition and Vaccination scenario), risks associated with major non-communicable diseases (NCDs; Improved Behavioural and Metabolic Risks scenario), and the combined effects of these three scenarios. Using the Shared Socioeconomic Pathways climate scenarios SSP2-4.5 as reference and SSP1-1.9 as an optimistic alternative in the Safer Environment scenario, we accounted for climate change impact on health by using the most recent Intergovernmental Panel on Climate Change temperature forecasts and published trajectories of ambient air pollution for the same two scenarios. Life expectancy and healthy life expectancy were computed using standard methods. The forecasting framework includes computing the age-sex-specific future population for each location and separately for each scenario. 95% uncertainty intervals (UIs) for each individual future estimate were derived from the 2·5th and 97·5th percentiles of distributions generated from propagating 500 draws through the multistage computational pipeline. Findings: In the reference scenario forecast, global and super-regional life expectancy increased from 2022 to 2050, but improvement was at a slower pace than in the three decades preceding the COVID-19 pandemic (beginning in 2020). Gains in future life expectancy were forecasted to be greatest in super-regions with comparatively low life expectancies (such as sub-Saharan Africa) compared with super-regions with higher life expectancies (such as the high-income super-region), leading to a trend towards convergence in life expectancy across locations between now and 2050. At the super-region level, forecasted healthy life expectancy patterns were similar to those of life expectancies. Forecasts for the reference scenario found that health will improve in the coming decades, with all-cause age-standardised DALY rates decreasing in every GBD super-region. The total DALY burden measured in counts, however, will increase in every super-region, largely a function of population ageing and growth. We also forecasted that both DALY counts and age-standardised DALY rates will continue to shift from CMNNs to NCDs, with the most pronounced shifts occurring in sub-Saharan Africa (60·1% [95% UI 56·8–63·1] of DALYs were from CMNNs in 2022 compared with 35·8% [31·0–45·0] in 2050) and south Asia (31·7% [29·2–34·1] to 15·5% [13·7–17·5]). This shift is reflected in the leading global causes of DALYs, with the top four causes in 2050 being ischaemic heart disease, stroke, diabetes, and chronic obstructive pulmonary disease, compared with 2022, with ischaemic heart disease, neonatal disorders, stroke, and lower respiratory infections at the top. The global proportion of DALYs due to YLDs likewise increased from 33·8% (27·4–40·3) to 41·1% (33·9–48·1) from 2022 to 2050, demonstrating an important shift in overall disease burden towards morbidity and away from premature death. The largest shift of this kind was forecasted for sub-Saharan Africa, from 20·1% (15·6–25·3) of DALYs due to YLDs in 2022 to 35·6% (26·5–43·0) in 2050. In the assessment of alternative future scenarios, the combined effects of the scenarios (Safer Environment, Improved Childhood Nutrition and Vaccination, and Improved Behavioural and Metabolic Risks scenarios) demonstrated an important decrease in the global burden of DALYs in 2050 of 15·4% (13·5–17·5) compared with the reference scenario, with decreases across super-regions ranging from 10·4% (9·7–11·3) in the high-income super-region to 23·9% (20·7–27·3) in north Africa and the Middle East. The Safer Environment scenario had its largest decrease in sub-Saharan Africa (5·2% [3·5–6·8]), the Improved Behavioural and Metabolic Risks scenario in north Africa and the Middle East (23·2% [20·2–26·5]), and the Improved Nutrition and Vaccination scenario in sub-Saharan Africa (2·0% [–0·6 to 3·6]). Interpretation: Globally, life expectancy and age-standardised disease burden were forecasted to improve between 2022 and 2050, with the majority of the burden continuing to shift from CMNNs to NCDs. That said, continued progress on reducing the CMNN disease burden will be dependent on maintaining investment in and policy emphasis on CMNN disease prevention and treatment. Mostly due to growth and ageing of populations, the number of deaths and DALYs due to all causes combined will generally increase. By constructing alternative future scenarios wherein certain risk exposures are eliminated by 2050, we have shown that opportunities exist to substantially improve health outcomes in the future through concerted efforts to prevent exposure to well established risk factors and to expand access to key health interventions