LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructions

Large language models (LLMs) with instruction fine-tuning demonstrate superior generative capabilities. However, these models are resource-intensive. To alleviate this issue, we explore distilling knowledge from instruction-tuned LLMs into much smaller ones. To this end, we carefully develop a large set of 2.58M instructions based on both existing and newly-generated instructions. In addition to being sizable, we design our instructions to cover a broad set of topics to ensure diversity. Extensive analysis of our instruction dataset confirms its diversity, and we generate responses for these instructions using gpt-3.5-turbo. Leveraging these instructions, we fine-tune a diverse herd of models, collectively referred to as LaMini-LM, which includes models from both the encoder-decoder and decoder-only families, with varying sizes. We evaluate the performance of our models using automatic metrics on 15 different natural language processing (NLP) benchmarks, as well as through human assessment. The results demonstrate that our proposed LaMini-LM models are comparable to competitive baselines, while being much smaller in size.Comment: 21 pages, 8 figures, 17 tables, accepted by EACL2024 main conferenc

Understanding In-Context Learning from Repetitions

This paper explores the elusive mechanism underpinning in-context learning in Large Language Models (LLMs). Our work provides a novel perspective by examining in-context learning via the lens of surface repetitions. We quantitatively investigate the role of surface features in text generation, and empirically establish the existence of \emph{token co-occurrence reinforcement}, a principle that strengthens the relationship between two tokens based on their contextual co-occurrences. By investigating the dual impacts of these features, our research illuminates the internal workings of in-context learning and expounds on the reasons for its failures. This paper provides an essential contribution to the understanding of in-context learning and its potential limitations, providing a fresh perspective on this exciting capability

Research on the Heating of Deicing Fluid in a New Reshaped Coiled Tube

Aircraft ground deicing operation is significant to ensure civil flight safety in winter. Helically coiled tube is the important heat exchanger in Chinese deicing fluid heating system. In order to improve the deicing efficiency, the research focuses on heat transfer enhancement of deicing fluid in the tube. Based on the field synergy principle, a new reshaped tube (TCHC) is designed by ring-rib convex on the inner wall. Deicing fluid is high viscosity ethylene-glycol-based mixture. Because of the power function relation between high viscosity and temperature, viscosity has a negative influence on heat transfer. The number of ring-ribs and inlet velocity are two key parameters to the heat transfer performance. For both water and ethylene glycol, the outlet temperature rises when the number of ring-ribs increases to a certain limit. However, the increasing of velocity reduces heating time, which results in lower outlet temperature. The heating experiment of the original tube is conducted. The error between experiment and simulation is less than 5%. The outlet temperature of TCHC increases by 3.76%. As a result, TCHC efficiently promotes the coordination of velocity and temperature fields by changing the velocity field. TCHC has enhanced heat transfer of high viscosity deicing fluid

A preliminary analysis of the effect of the new rural cooperative medical scheme on inpatient care at a county hospital

BACKGROUND: China in 2009 committed to reach universal health coverage by promoting three forms of health insurance; NCMS for the rural population, UEBMI for formally employed urban residents and URBMI for other urban residents. NCMS has expanded to near universal coverage in rural China since launching in 2003. The objective of this study aimed to assess the effect of NCMS on inpatient care utilization from 2003 to 2012 at Longyou county hospital, Zhejiang province. METHODS: The research was conducted at Longyou county, Zhejiang province. All registered inpatient admissions from January 1, 2003, to June 30, 2012, were included in the study. The PLSQL Developer software was used to select the interesting variables in the hospital information database and saved in an Excel 2003 file. The interesting variables included the patients’ general information (name, gender, age, payment method), discharge diagnosis, length of hospital stay, and expenditure (total expenditure and out-of-pocket payment). Two common diseases (coronary arteriosclerotic disease and pneumonia) were selected as tracer conditions. RESULTS: 292,400 rural residents were enrolled in the Longyou county NCMS by 2011, 95.4% of the eligible population. A total of 145,744 inpatient admissions were registered from 1 January 2003 to 30 June 2012. The proportion of inpatients covered by NCMS increased from 30.3% in 2004 to 54.2% in 2012 while the proportion of inpatients covered by UEBMI increased from 7.7% in 2003 to 14.7% in 2012. The average expenditure for UEBMI insured inpatients was higher than the average for NCMS insured inpatients, although the gap was narrowing. The average length of hospital stay increased every year for all inpatients, but was higher for UEBMI inpatients than for NCMS insured inpatients. For both tracer conditions the results were similar to the above findings. CONCLUSIONS: NCMS has improved coverage height for its enrollees and resulted in increased cost of care per inpatient admission at the county hospital. However, wide differences persist between the two insurance systems in coverage height. Both systems are associated with increasing lengths of stay and rising cost per inpatient admission. We found that around 30% of inpatients were not covered by any of the two public health insurance systems, which calls for further studies

Strengths and Weaknesses of Immunotherapy for Advanced Non-Small-Cell Lung Cancer: A Meta-Analysis of 12 Randomized Controlled Trials

Background: Lung cancer is one of the leading causes of cancer death worldwide. Non-small-cell lung cancer (NSCLC) accounts for approximately 85 % of all lung cancers. Immunotherapy has yielded no consistent benefit to date for those patients. Assessing the objective efficacy and safety of immunotherapy for advanced NSCLC patients will help to instruct the future development of immunotherapeutic drugs. Methodology and Principal Findings: We performed a meta-analysis of 12 randomized controlled trials including 3134 patients (1570 patients in the immunotherapy group and 1564 patients in the control group) with histologically confirmed stage IIIA, IIIB, or IV NSCLC. The analysis was executed with efficacy end points regarding overall survival (OS), progressionfre

Lignosulfonic Acid Exhibits Broadly Anti-HIV-1 Activity – Potential as a Microbicide Candidate for the Prevention of HIV-1 Sexual Transmission

Some secondary metabolites from plants show to have potent inhibitory activities against microbial pathogens, such as human immunodeficiency virus (HIV), herpes simplex virus (HSV), Treponema pallidum, Neisseria gonorrhoeae, etc. Here we report that lignosulfonic acid (LSA), a polymeric lignin derivative, exhibits potent and broad activity against HIV-1 isolates of diverse subtypes including two North America strains and a number of Chinese clinical isolates values ranging from 21.4 to 633 nM. Distinct from other polyanions, LSA functions as an entry inhibitor with multiple targets on viral gp120 as well as on host receptor CD4 and co-receptors CCR5/CXCR4. LSA blocks viral entry as determined by time-of-drug addiction and cell-cell fusion assays. Moreover, LSA inhibits CD4-gp120 interaction by blocking the binding of antibodies specific for CD4-binding sites (CD4bs) and for the V3 loop of gp120. Similarly, LSA interacts with CCR5 and CXCR4 via its inhibition of specific anti-CCR5 and anti-CXCR4 antibodies, respectively. Interestingly, the combination of LSA with AZT and Nevirapine exhibits synergism in viral inhibition. For the purpose of microbicide development, LSA displays low in vitro cytotoxicity to human genital tract epithelial cells, does not stimulate NF-κB activation and has no significant up-regulation of IL-1α/β and IL-8 as compared with N-9. Lastly, LSA shows no adverse effect on the epithelial integrity and the junctional protein expression. Taken together, our findings suggest that LSA can be a potential candidate for tropical microbicide

Co-Evolution of Primate SAMHD1 and Lentivirus Vpx Leads to the Loss of the vpx Gene in HIV-1 Ancestor

Cross-species transmission and adaptation of simian immunodeficiency viruses (SIVs) to humans have given rise to human immunodeficiency viruses (HIVs). HIV type 1 (HIV-1) and type 2 (HIV-2) were derived from SIVs that infected chimpanzee (SIVcpz) and sooty mangabey (SIVsm), respectively. The HIV-1 restriction factor SAMHD1 inhibits HIV-1 infection in human myeloid cells and can be counteracted by the Vpx protein of HIV-2 and the SIVsm lineage. However, HIV-1 and its ancestor SIVcpz do not encode a Vpx protein and HIV-1 has not evolved a mechanism to overcome SAMHD1-mediated restriction. Here we show that the co-evolution of primate SAMHD1 and lentivirus Vpx leads to the loss of the vpx gene in SIVcpz and HIV-1. We found evidence for positive selection of SAMHD1 in orangutan, gibbon, rhesus macaque, and marmoset, but not in human, chimpanzee and gorilla that are natural hosts of Vpx-negative HIV-1, SIVcpz and SIVgor, respectively, indicating that vpx drives the evolution of primate SAMHD1. Ancestral host state reconstruction and temporal dynamic analyses suggest that the most recent common ancestor of SIVrcm, SIVmnd, SIVcpz, SIVgor and HIV-1 was a SIV that had a vpx gene; however, the vpx gene of SIVcpz was lost approximately 3643 to 2969 years ago during the infection of chimpanzees. Thus, HIV-1 could not inherit the lost vpx gene from its ancestor SIVcpz. The lack of Vpx in HIV-1 results in restricted infection in myeloid cells that are important for antiviral immunity, which could contribute to the AIDS pandemic by escaping the immune responses