25 research outputs found
Knowledge Distillation Label Smoothing: Fact or Fallacy?
Originally proposed as a method for knowledge transfer from one model to
another, some recent studies have suggested that knowledge distillation (KD) is
in fact a form of regularization. Perhaps the strongest argument of all for
this new perspective comes from its apparent similarities with label smoothing
(LS). Here we re-examine this stated equivalence between the two methods by
comparing the predictive confidences of the models they train. Experiments on
four text classification tasks involving models of different sizes show that:
(a) In most settings, KD and LS drive model confidence in completely opposite
directions, and (b) In KD, the student inherits not only its knowledge but also
its confidence from the teacher, reinforcing the classical knowledge transfer
view.Comment: EMNLP 202
Learning Cross-Lingual IR from an English Retriever
We present DR.DECR (Dense Retrieval with Distillation-Enhanced Cross-Lingual
Representation), a new cross-lingual information retrieval (CLIR) system
trained using multi-stage knowledge distillation (KD). The teacher of DR.DECR
relies on a highly effective but computationally expensive two-stage inference
process consisting of query translation and monolingual IR, while the student,
DR.DECR, executes a single CLIR step. We teach DR.DECR powerful multilingual
representations as well as CLIR by optimizing two corresponding KD objectives.
Learning useful representations of non-English text from an English-only
retriever is accomplished through a cross-lingual token alignment algorithm
that relies on the representation capabilities of the underlying multilingual
encoders. In both in-domain and zero-shot out-of-domain evaluation, DR.DECR
demonstrates far superior accuracy over direct fine-tuning with labeled CLIR
data. It is also the best single-model retriever on the XOR-TyDi benchmark at
the time of this writing.Comment: 9 page
A Fuzzy ANP Based Grey Relational Approach to Evaluate CRM System in Context of Bangladesh
This study aims to select a suitable CRM (customer relationship management) system among different possible alternatives for organization’s in Bangladesh. Since, evaluating CRM system on the basis of lot of attributes leads us to Multiple-criteria decision analysis (MCDA) problems. In this study, a hybrid MCDA models were used. FuzzyANP (Analytic Network Process) and GRA (Grey Relational Analysis) approaches were adopted to solve the problem. The study explored that the Hubspot CRM was optimal solution in context of Bangladesh. Our research will beneficial to the organizing for better customer support. As far our knowledge goes, this is the first attempt to select CRM softwares in context of Bangladesh. Keywords: Analytic network process; Customer relationship management system; Grey relational analysis; Multiple-criteria decision analysis DOI: 10.7176/IKM/11-4-06 Publication date:June 30th 202
UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers
Many information retrieval tasks require large labeled datasets for
fine-tuning. However, such datasets are often unavailable, and their utility
for real-world applications can diminish quickly due to domain shifts. To
address this challenge, we develop and motivate a method for using large
language models (LLMs) to generate large numbers of synthetic queries cheaply.
The method begins by generating a small number of synthetic queries using an
expensive LLM. After that, a much less expensive one is used to create large
numbers of synthetic queries, which are used to fine-tune a family of reranker
models. These rerankers are then distilled into a single efficient retriever
for use in the target domain. We show that this technique boosts zero-shot
accuracy in long-tail domains and achieves substantially lower latency than
standard reranking methods.Comment: Long Paper at Empirical Methods in Natural Language Processing
(EMNLP) 202
Ensemble-Instruct: Generating Instruction-Tuning Data with a Heterogeneous Mixture of LMs
Using in-context learning (ICL) for data generation, techniques such as
Self-Instruct (Wang et al., 2023) or the follow-up Alpaca (Taori et al., 2023)
can train strong conversational agents with only a small amount of human
supervision. One limitation of these approaches is that they resort to very
large language models (around 175B parameters) that are also proprietary and
non-public. Here we explore the application of such techniques to language
models that are much smaller (around 10B--40B parameters) and have permissive
licenses. We find the Self-Instruct approach to be less effective at these
sizes and propose new ICL methods that draw on two main ideas: (a)
Categorization and simplification of the ICL templates to make prompt learning
easier for the LM, and (b) Ensembling over multiple LM outputs to help select
high-quality synthetic examples. Our algorithm leverages the 175 Self-Instruct
seed tasks and employs separate pipelines for instructions that require an
input and instructions that do not. Empirical investigations with different LMs
show that: (1) Our proposed method yields higher-quality instruction tuning
data than Self-Instruct, (2) It improves performances of both vanilla and
instruction-tuned LMs by significant margins, and (3) Smaller instruction-tuned
LMs generate more useful outputs than their larger un-tuned counterparts. Our
codebase is available at https://github.com/IBM/ensemble-instruct
PrimeQA: The Prime Repository for State-of-the-Art Multilingual Question Answering Research and Development
The field of Question Answering (QA) has made remarkable progress in recent
years, thanks to the advent of large pre-trained language models, newer
realistic benchmark datasets with leaderboards, and novel algorithms for key
components such as retrievers and readers. In this paper, we introduce PRIMEQA:
a one-stop and open-source QA repository with an aim to democratize QA
re-search and facilitate easy replication of state-of-the-art (SOTA) QA
methods. PRIMEQA supports core QA functionalities like retrieval and reading
comprehension as well as auxiliary capabilities such as question generation.It
has been designed as an end-to-end toolkit for various use cases: building
front-end applications, replicating SOTA methods on pub-lic benchmarks, and
expanding pre-existing methods. PRIMEQA is available at :
https://github.com/primeqa
Recommended from our members
Global burden of 288 causes of death and life expectancy decomposition in 204 countries and territories and 811 subnational locations, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021
BACKGROUND Regular, detailed reporting on population health by underlying cause of death is fundamental for public health decision making. Cause-specific estimates of mortality and the subsequent effects on life expectancy worldwide are valuable metrics to gauge progress in reducing mortality rates. These estimates are particularly important following large-scale mortality spikes, such as the COVID-19 pandemic. When systematically analysed, mortality rates and life expectancy allow comparisons of the consequences of causes of death globally and over time, providing a nuanced understanding of the effect of these causes on global populations. METHODS The Global Burden of Diseases, Injuries, and Risk Factors Study (GBD) 2021 cause-of-death analysis estimated mortality and years of life lost (YLLs) from 288 causes of death by age-sex-location-year in 204 countries and territories and 811 subnational locations for each year from 1990 until 2021. The analysis used 56 604 data sources, including data from vital registration and verbal autopsy as well as surveys, censuses, surveillance systems, and cancer registries, among others. As with previous GBD rounds, cause-specific death rates for most causes were estimated using the Cause of Death Ensemble model-a modelling tool developed for GBD to assess the out-of-sample predictive validity of different statistical models and covariate permutations and combine those results to produce cause-specific mortality estimates-with alternative strategies adapted to model causes with insufficient data, substantial changes in reporting over the study period, or unusual epidemiology. YLLs were computed as the product of the number of deaths for each cause-age-sex-location-year and the standard life expectancy at each age. As part of the modelling process, uncertainty intervals (UIs) were generated using the 2·5th and 97·5th percentiles from a 1000-draw distribution for each metric. We decomposed life expectancy by cause of death, location, and year to show cause-specific effects on life expectancy from 1990 to 2021. We also used the coefficient of variation and the fraction of population affected by 90% of deaths to highlight concentrations of mortality. Findings are reported in counts and age-standardised rates. Methodological improvements for cause-of-death estimates in GBD 2021 include the expansion of under-5-years age group to include four new age groups, enhanced methods to account for stochastic variation of sparse data, and the inclusion of COVID-19 and other pandemic-related mortality-which includes excess mortality associated with the pandemic, excluding COVID-19, lower respiratory infections, measles, malaria, and pertussis. For this analysis, 199 new country-years of vital registration cause-of-death data, 5 country-years of surveillance data, 21 country-years of verbal autopsy data, and 94 country-years of other data types were added to those used in previous GBD rounds. FINDINGS The leading causes of age-standardised deaths globally were the same in 2019 as they were in 1990; in descending order, these were, ischaemic heart disease, stroke, chronic obstructive pulmonary disease, and lower respiratory infections. In 2021, however, COVID-19 replaced stroke as the second-leading age-standardised cause of death, with 94·0 deaths (95% UI 89·2-100·0) per 100 000 population. The COVID-19 pandemic shifted the rankings of the leading five causes, lowering stroke to the third-leading and chronic obstructive pulmonary disease to the fourth-leading position. In 2021, the highest age-standardised death rates from COVID-19 occurred in sub-Saharan Africa (271·0 deaths [250·1-290·7] per 100 000 population) and Latin America and the Caribbean (195·4 deaths [182·1-211·4] per 100 000 population). The lowest age-standardised death rates from COVID-19 were in the high-income super-region (48·1 deaths [47·4-48·8] per 100 000 population) and southeast Asia, east Asia, and Oceania (23·2 deaths [16·3-37·2] per 100 000 population). Globally, life expectancy steadily improved between 1990 and 2019 for 18 of the 22 investigated causes. Decomposition of global and regional life expectancy showed the positive effect that reductions in deaths from enteric infections, lower respiratory infections, stroke, and neonatal deaths, among others have contributed to improved survival over the study period. However, a net reduction of 1·6 years occurred in global life expectancy between 2019 and 2021, primarily due to increased death rates from COVID-19 and other pandemic-related mortality. Life expectancy was highly variable between super-regions over the study period, with southeast Asia, east Asia, and Oceania gaining 8·3 years (6·7-9·9) overall, while having the smallest reduction in life expectancy due to COVID-19 (0·4 years). The largest reduction in life expectancy due to COVID-19 occurred in Latin America and the Caribbean (3·6 years). Additionally, 53 of the 288 causes of death were highly concentrated in locations with less than 50% of the global population as of 2021, and these causes of death became progressively more concentrated since 1990, when only 44 causes showed this pattern. The concentration phenomenon is discussed heuristically with respect to enteric and lower respiratory infections, malaria, HIV/AIDS, neonatal disorders, tuberculosis, and measles. INTERPRETATION Long-standing gains in life expectancy and reductions in many of the leading causes of death have been disrupted by the COVID-19 pandemic, the adverse effects of which were spread unevenly among populations. Despite the pandemic, there has been continued progress in combatting several notable causes of death, leading to improved global life expectancy over the study period. Each of the seven GBD super-regions showed an overall improvement from 1990 and 2021, obscuring the negative effect in the years of the pandemic. Additionally, our findings regarding regional variation in causes of death driving increases in life expectancy hold clear policy utility. Analyses of shifting mortality trends reveal that several causes, once widespread globally, are now increasingly concentrated geographically. These changes in mortality concentration, alongside further investigation of changing risks, interventions, and relevant policy, present an important opportunity to deepen our understanding of mortality-reduction strategies. Examining patterns in mortality concentration might reveal areas where successful public health interventions have been implemented. Translating these successes to locations where certain causes of death remain entrenched can inform policies that work to improve life expectancy for people everywhere. FUNDING Bill & Melinda Gates Foundation
Mortality from gastrointestinal congenital anomalies at 264 hospitals in 74 low-income, middle-income, and high-income countries: a multicentre, international, prospective cohort study
Summary
Background Congenital anomalies are the fifth leading cause of mortality in children younger than 5 years globally.
Many gastrointestinal congenital anomalies are fatal without timely access to neonatal surgical care, but few studies
have been done on these conditions in low-income and middle-income countries (LMICs). We compared outcomes of
the seven most common gastrointestinal congenital anomalies in low-income, middle-income, and high-income
countries globally, and identified factors associated with mortality.
Methods We did a multicentre, international prospective cohort study of patients younger than 16 years, presenting to
hospital for the first time with oesophageal atresia, congenital diaphragmatic hernia, intestinal atresia, gastroschisis,
exomphalos, anorectal malformation, and Hirschsprung’s disease. Recruitment was of consecutive patients for a
minimum of 1 month between October, 2018, and April, 2019. We collected data on patient demographics, clinical
status, interventions, and outcomes using the REDCap platform. Patients were followed up for 30 days after primary
intervention, or 30 days after admission if they did not receive an intervention. The primary outcome was all-cause,
in-hospital mortality for all conditions combined and each condition individually, stratified by country income status.
We did a complete case analysis.
Findings We included 3849 patients with 3975 study conditions (560 with oesophageal atresia, 448 with congenital
diaphragmatic hernia, 681 with intestinal atresia, 453 with gastroschisis, 325 with exomphalos, 991 with anorectal
malformation, and 517 with Hirschsprung’s disease) from 264 hospitals (89 in high-income countries, 166 in middleincome
countries, and nine in low-income countries) in 74 countries. Of the 3849 patients, 2231 (58·0%) were male.
Median gestational age at birth was 38 weeks (IQR 36–39) and median bodyweight at presentation was 2·8 kg (2·3–3·3).
Mortality among all patients was 37 (39·8%) of 93 in low-income countries, 583 (20·4%) of 2860 in middle-income
countries, and 50 (5·6%) of 896 in high-income countries (p<0·0001 between all country income groups).
Gastroschisis had the greatest difference in mortality between country income strata (nine [90·0%] of ten in lowincome
countries, 97 [31·9%] of 304 in middle-income countries, and two [1·4%] of 139 in high-income countries;
p≤0·0001 between all country income groups). Factors significantly associated with higher mortality for all patients
combined included country income status (low-income vs high-income countries, risk ratio 2·78 [95% CI 1·88–4·11],
p<0·0001; middle-income vs high-income countries, 2·11 [1·59–2·79], p<0·0001), sepsis at presentation (1·20
[1·04–1·40], p=0·016), higher American Society of Anesthesiologists (ASA) score at primary intervention
(ASA 4–5 vs ASA 1–2, 1·82 [1·40–2·35], p<0·0001; ASA 3 vs ASA 1–2, 1·58, [1·30–1·92], p<0·0001]), surgical safety
checklist not used (1·39 [1·02–1·90], p=0·035), and ventilation or parenteral nutrition unavailable when needed
(ventilation 1·96, [1·41–2·71], p=0·0001; parenteral nutrition 1·35, [1·05–1·74], p=0·018). Administration of
parenteral nutrition (0·61, [0·47–0·79], p=0·0002) and use of a peripherally inserted central catheter (0·65
[0·50–0·86], p=0·0024) or percutaneous central line (0·69 [0·48–1·00], p=0·049) were associated with lower mortality.
Interpretation Unacceptable differences in mortality exist for gastrointestinal congenital anomalies between lowincome,
middle-income, and high-income countries. Improving access to quality neonatal surgical care in LMICs will
be vital to achieve Sustainable Development Goal 3.2 of ending preventable deaths in neonates and children younger
than 5 years by 2030