OpenDelta: A Plug-and-play Library for Parameter-efficient Adaptation of Pre-trained Models

The scale of large pre-trained models (PTMs) poses significant challenges in adapting to downstream tasks due to the high optimization overhead and storage costs associated with full-parameter fine-tuning. To address this, many studies explore parameter-efficient tuning methods, also framed as "delta tuning", which updates only a small subset of parameters, known as "delta modules", while keeping the backbone model's parameters fixed. However, the practicality and flexibility of delta tuning have been limited due to existing implementations that directly modify the code of the backbone PTMs and hard-code specific delta tuning methods for each PTM. In this paper, we present OpenDelta, an open-source library that overcomes these limitations by providing a plug-and-play implementation of various delta tuning methods. Our novel techniques eliminate the need to modify the backbone PTMs' code, making OpenDelta compatible with different, even novel PTMs. OpenDelta is designed to be simple, modular, and extensible, providing a comprehensive platform for researchers and practitioners to adapt large PTMs efficiently.Comment: Accepted to ACL 2023 Demo trac

SARS-CoV-2-Specific Adaptive Immunity in COVID-19 Survivors With Asthma

BackgroundAsthma patients potentially have impaired adaptive immunity to virus infection. The levels of SARS-CoV-2-specific adaptive immunity between COVID-19 survivors with and without asthma are presently unclear.MethodsCOVID-19 survivors (patients with asthma n=11, with allergies n=8, and COVID-19 only n=17) and non-COVID-19 individuals (asthmatic patients n=10 and healthy controls n=9) were included. The COVID-19 patients were followed up at about 8 months and 16 months after discharge. The clinical characteristics, lymphocyte subsets, memory T cells, and humoral immunity including SARS-CoV-2 specific antibodies, SARS-CoV-2 pseudotyped virus neutralization assay, and memory B cells were analyzed in these subjects.ResultsThe strength of virus-specific T cell response in COVID-19 survivors was positively correlated with the percentage of blood eosinophils and Treg cells (r=0.4007, p=0.0188; and r=0.4435, p=0.0086 respectively) at 8-month follow-up. There were no statistical differences in the levels of SARS-CoV-2-specific T cell response between the COVID-19 survivors with, and without, asthma. Compared to those without asthma, the COVID-19 with asthma survivors had higher levels of SARS-CoV-2-specific neutralizing antibodies (NAbs) at the 8-month follow-up (p<0.05). Moreover, the level of NAbs in COVID-19 survivors was positively correlated with the percentage of Treg and cTfh2 cells (r=0.5037, p=0.002; and r=0.4846, p=0.0141), and negatively correlated with the percentage of Th1 and Th17 cells (r=-0.5701, p=0.0003; and r=-0.3656, p=0.0308), the ratio of Th1/Th2, Th17/Treg, and cTfh1/cTfh2 cell (r=-0.5356, r=-0.5947, r=-0.4485; all p<0.05). The decay rate of NAbs in the COVID-19 survivors with asthma was not significantly different from that of those without asthma at 16-month follow-up.ConclusionThe level of SARS-CoV-2-specific NAbs in COVID-19 survivors with asthma was higher than that of those without asthma at 8-month follow-up. The SARS-CoV-2-specific T cell immunity was associated with blood eosinophils and Treg percentages. The SARS-CoV-2-specific humoral immunity was closely associated with cTfh2/cTfh1 imbalance and Treg/Th17 ratio. According to the findings, asthmatic patients in COVID-19 convalescent period may benefit from an enhanced specific humoral immunity, which associates with skewed Th2/Th1 and Treg/Th17 immune

Sparse Structure Search for Parameter-Efficient Tuning

Adapting large pre-trained models (PTMs) through fine-tuning imposes prohibitive computational and storage burdens. Recent studies of parameter-efficient tuning (PET) find that only optimizing a small portion of parameters conditioned on PTMs could yield on-par performance compared to conventional fine-tuning. Generally, PET methods exquisitely design parameter-efficient modules (PET modules) which could be applied to arbitrary fine-grained positions inside PTMs. However, the effectiveness of these fine-grained positions largely relies on sophisticated manual designation, thereby usually producing sub-optimal results. In contrast to the manual designation, we explore constructing PET modules in an automatic manner. We automatically \textbf{S}earch for the \textbf{S}parse \textbf{S}tructure of \textbf{P}arameter-\textbf{E}fficient \textbf{T}uning (S$^3$PET). Based on a unified framework of various PET methods, S$^3$PET conducts the differentiable PET structure search through bi-level optimization and proposes shifted global sigmoid method to explicitly control the number of trainable parameters. Extensive experiments show that S$^3$PET surpasses manual and random structures with less trainable parameters. The searched structures preserve more than 99\% fine-tuning performance with 0.01\% trainable parameters. Moreover, the advantage of S$^3$PET is amplified with extremely low trainable parameters budgets (0.0009\%$\sim$0.01\%). The searched structures are transferable and explainable, providing suggestions and guidance for the future design of PET methods

Multiple linear regression analysis of the factors related with miR-199a-5p expression.

<p>Multiple linear regression analysis of the factors related with miR-199a-5p expression.</p

Novel imaging phenotypes of naïve asthma patients with distinctive clinical characteristics and T2 inflammation traits

Objective:This study aims to describe the imaging features of naïve asthma patients, defined as not receiving corticosteroids or other asthma medications for at least 1 month, and their association with therapeutic response, and to discover novel unbiased imaging phenotypes.Methods:A total of 109 naïve asthma patients and 50 healthy controls were enrolled in this study. Clinical data and imaging indices of high-resolution computed tomography were collected. The correlation between imaging indices and clinical features was analyzed. Cluster analyses were adopted to determine three novel imaging phenotypes.Results:Compared with healthy controls, naïve asthma patients presented higher scores of airway remodeling, bronchiectasis, and mucus plugs. Mean airway wall area (WA)% was inversely correlated with mid-expiratory flow velocity% predicted. The extent score of bronchiectasis was positively correlated with smoking history and significantly increased in the high mucus group. Mucus plugs were related to improving lung function and type 2 (T2) inflammation, as assessed by sputum and blood eosinophils and fraction of exhaled nitric oxide. Cluster 1 patients had a high proportion of emphysema, the best lung function, and the lowest T2 inflammation; cluster 2 patients had severe airway remodeling, relatively good lung function, and moderate T2 inflammation; cluster 3 patients had severe airway remodeling, mucus plugs, and bronchiectasis, and showed the worst lung function and highest T2 inflammation.Conclusion:Naïve asthma patients had the imaging traits of airway remodeling, bronchiectasis, and mucus plugs. The unbiased imaging phenotypes had good consistency with clinical characteristics, therapeutic response, and T2 inflammation expression in naïve asthma patients

miR-199a-5p expression levels from different cells following LPS stimulation.

<p>(A-C) miR-199a-5p expression levels in PB-derived macrophages (A), lymphocytes (B) and neutrophils (C) isolated from three healthy subjects treated or untreated with LPS (100 ng/mL) for 24 hours. (D) miR-199a-5p expression levels in BEAS-2B cell lines, untreated or treated with LPS (100 ng/mL) for 24 hours. Relative miR-199a-5p levels were calculated over untreated controls.</p

Clinical characteristics of participants.

<p>Clinical characteristics of participants.</p

In silico analysis into the functional role of miRNA.

<p>Overview of the GO(A) and KEGG(B) pathways that contain at least 1 of the genes targeted by miR-199a-5p. Target genes are indicated on both sides. Targets expressed in the pathway are indicated in blue, and white indicates that the gene was not expressed in the gene set. Arrows represent pathways with potential interest in the proliferation of smooth muscle cells.</p

Relative miR-199a-5p levels in sputum in different groups.

<p>(A) Normalized miR-199a-5p expression levels in sputum from different inflammatory phenotypes and healthy controls. (B) The correlation between the miR-199a-5p expression levels in plasma and sputum. Data are presented as dot plots with fitted regression lines. The miR-199a-5p expression levels were normalized to U6. Spearman R-value and p-value are indicated.</p