FederatedScope-LLM: A Comprehensive Package for Fine-tuning Large Language Models in Federated Learning

LLMs have demonstrated great capabilities in various NLP tasks. Different entities can further improve the performance of those LLMs on their specific downstream tasks by fine-tuning LLMs. When several entities have similar interested tasks, but their data cannot be shared because of privacy concerns regulations, federated learning (FL) is a mainstream solution to leverage the data of different entities. However, fine-tuning LLMs in federated learning settings still lacks adequate support from existing FL frameworks because it has to deal with optimizing the consumption of significant communication and computational resources, data preparation for different tasks, and distinct information protection demands. This paper first discusses these challenges of federated fine-tuning LLMs, and introduces our package FS-LLM as a main contribution, which consists of the following components: (1) we build an end-to-end benchmarking pipeline, automizing the processes of dataset preprocessing, federated fine-tuning execution, and performance evaluation on federated LLM fine-tuning; (2) we provide comprehensive federated parameter-efficient fine-tuning algorithm implementations and versatile programming interfaces for future extension in FL scenarios with low communication and computation costs, even without accessing the full model; (3) we adopt several accelerating and resource-efficient operators for fine-tuning LLMs with limited resources and the flexible pluggable sub-routines for interdisciplinary study. We conduct extensive experiments to validate the effectiveness of FS-LLM and benchmark advanced LLMs with state-of-the-art parameter-efficient fine-tuning algorithms in FL settings, which also yields valuable insights into federated fine-tuning LLMs for the research community. To facilitate further research and adoption, we release FS-LLM at https://github.com/alibaba/FederatedScope/tree/llm.Comment: Source code: https://github.com/alibaba/FederatedScope/tree/ll

Physiological effects of combined NaCl and NaHCO3 stress on the seedlings of two maple species

Salt stress impacts growth and physiological processes in plants, and some plants exposed to salt stress will produce physiological mechanisms to adapt to the new environment. However, the effects of combined NaCl and NaHCO3 stress on the seedlings of Acer species are understudied. In this study, we designed an experiment to measure physiological characteristics by establishing a range of NaCl and NaHCO3 concentrations (0, 25, 50, 75, and 100 mmol L-1) to estimate the compound salt tolerance of Acer ginnala and Acer palmatum. When the concentrations of NaCl and NaHCO3 were 25 mmol L-1, the leaf water content, relative conductivity, malondialdehyde (MDA) content, proline content, soluble sugar content, and chlorophyll did not change (p > 0.05) in two maple seedlings. At concentrations greater than 50 mmol L-1, the relative conductivity and MDA content increased, proline and soluble sugars accumulated, and the potential activity of PS II (Fv/Fo), potential photochemical efficiency of PS II (Fv/Fm), PS II actual photochemical efficiency (Yield), and photosynthetic electron transfer efficiency (ETR) decreased (p < 0.05). The superoxide dismutase (SOD) and catalase (CAT) activities showed the same trend of first increasing and then decreasing (p < 0.05). The peroxidase (POD) activity increased only when concentrations of NaCl and NaHCO3 were 100 mmol L-1, while there was no statistical difference between the other treatments and the control. Therefore, the two maple seedlings adjusted their osmotic balance and alleviated oxidative stress by accumulating proline, soluble sugars and increasing CAT and SOD activities. Further analysis showed that both species are salt tolerant and the salt tolerance of Acer ginnala is better than that of Acer palmatum

Effect of the Cationic Block Structure on the Characteristics of Sludge Flocs Formed by Charge Neutralization and Patching

In this study, a template copolymer (TPAA) of (3-Acrylamidopropyl) trimethylammonium chloride (AATPAC) and acrylamide (AM) was successfully synthesized though ultrasonic-initiated template copolymerization (UTP), using sodium polyacrylate (PAAS) as a template. TPAA was characterized by an evident cationic microblock structure which was observed through the analyses of the reactivity ratio, Fourier transform infrared spectroscopy (FTIR), 1H (13C) nuclear magnetic resonance spectroscopy (1H (13C) NMR), and thermogravimetry/differential scanning calorimetry (TG/DSC). The introduction of the template could improve the monomer (AATPAC) reactivity ratio and increase the length and amount of AATPAC segments. This novel cationic microblock structure extremely enhanced the ability of charge neutralization, patching, and bridging, thus improving the activated sludge flocculation performance. The experiments of floc formation, breakage, and regrowth revealed that the cationic microblock structure in the copolymer resulted in large and compact flocs, and these flocs had a rapid regrowth when broken. Finally, the larger and more compact flocs contributed to the formation of more channels and voids, and therefore the specific resistance to filtration (SRF) reached a minimum

HPLC–DAD Analysis, SFE-CO₂ Extraction, and Antibacterial Activity on Bioactive Compounds from <i>Mosla chinensis</i> Maxim

Mosla chinensis Maxim is an annual herb with many potential purposes in agricultural, industrial, and pharmaceutical fields. At present, the extract of the whole plant from M. chinensis has been proven to demonstrate antifungal, antioxidant, and anti-inflammatory activities. Previous studies focused on the enzyme pretreatment in hydrodistillation from M. chinensis. However, organic solvent or supercritical fluid carbon dioxide extraction (SFE-CO2) methods, which are commonly utilized in industry, have seldom been studied and cannot provide multiple evaluations of yield. In this work, we analysed compounds from M. chinensis by HPLC–DAD, discussed n-hexane extraction, and conducted further investigations on SFE-CO2 through the design of response surface methodology (RSM). The sample obtained from pilot-scale SFE-CO2 was also tested against nine kinds of microorganisms. Single-factor results revealed that the extraction rates from M. chinensis by steam distillation, n-hexane extraction, and SFE-CO2 were 1%, 2.09%, and 3.26%, respectively. RSM results showed a significant improvement in extraction rate through optimising pressure and time, and the interaction of both factors was more important than that of temperature–pressure and temperature–time. A pilot-scale test with an extraction rate of 3.34% indicated that the predicted RSM condition was operable. In addition, samples from the pilot-scale SFE-CO2 showed antibacterial effects against three previously unreported bacteria (Gardnerella vaginalis, methicillin-resistant Staphylococcus aureus, and Propionibacterium acnes). These results fill the gap in previous research and provide more information for the application and development of M. chinensis in the future

Effects of diabetes mellitus and glycemic traits on cardiovascular morpho-functional phenotypes

Abstract Background The effects of diabetes on the cardiac and aortic structure and function remain unclear. Detecting and intervening these variations early is crucial for the prevention and management of complications. Cardiovascular magnetic resonance imaging-derived traits are established endophenotypes and serve as precise, early-detection, noninvasive clinical risk biomarkers. We conducted a Mendelian randomization (MR) study to examine the association between two types of diabetes, four glycemic traits, and preclinical endophenotypes of cardiac and aortic structure and function. Methods Independent genetic variants significantly associated with type 1 diabetes, type 2 diabetes, fasting insulin (FIns), fasting glucose (FGlu), 2 h-glucose post-challenge (2hGlu), and glycated hemoglobin (HbA1c) were selected as instrumental variables. The 96 cardiovascular magnetic resonance imaging traits came from six independent genome-wide association studies. These traits serve as preclinical endophenotypes and offer an early indication of the structure and function of the four cardiac chambers and two aortic sections. The primary analysis was performed using MR with the inverse-variance weighted method. Confirmation was achieved through Steiger filtering and testing to determine the causal direction. Sensitivity analyses were conducted using the weighted median, MR-Egger, and MR-PRESSO methods. Additionally, multivariable MR was used to adjust for potential effects associated with body mass index. Results Genetic susceptibility to type 1 diabetes was associated with increased ascending aortic distensibility. Conversely, type 2 diabetes showed a correlation with a reduced diameter and areas of the ascending aorta, as well as decreased distensibility of the descending aorta. Genetically predicted higher levels of FGlu and HbA1c were correlated with a decrease in diameter and areas of the ascending aorta. Furthermore, higher 2hGlu levels predominantly showed association with a reduced diameter of both the ascending and descending aorta. Higher FIns levels corresponded to increased regional myocardial-wall thicknesses at end-diastole, global myocardial-wall thickness at end-diastole, and regional peak circumferential strain of the left ventricle. Conclusions This study provides evidence that diabetes and glycemic traits have a causal relationship with cardiac and aortic structural and functional remodeling, highlighting the importance of intensive glucose-lowering for primary prevention of cardiovascular diseases

Differentiation of Human Induced Pluripotent Stem Cells to Mammary-like Organoids

Human induced pluripotent stem cells (iPSCs) can give rise to multiple cell types and hold great promise in regenerative medicine and disease-modeling applications. We have developed a reliable two-step protocol to generate human mammary-like organoids from iPSCs. Non-neural ectoderm-cell-containing spheres, referred to as mEBs, were first differentiated and enriched from iPSCs using MammoCult medium. Gene expression profile analysis suggested that mammary gland function-associated signaling pathways were hallmarks of 10-day differentiated mEBs. We then generated mammary-like organoids from 10-day mEBs using 3D floating mixed gel culture and a three-stage differentiation procedure. These organoids expressed common breast tissue, luminal, and basal markers, including estrogen receptor, and could be induced to produce milk protein. These results demonstrate that human iPSCs can be directed in vitro toward mammary lineage differentiation. Our findings provide an iPSC-based model for studying regulation of normal mammary cell fate and function as well as breast disease development

Additional file 2: Figure S2. of Identification of EGF-NF-κB-FOXC1 signaling axis in basal-like breast cancer

Inhibition of NF-κB p65 affects FOXC1 protein levels. MDA-MB-468 cells were serum-starved overnight and treated with EGF for 24 h after pre-incubation with NF-κB inhibitors, Bay 11–7082 or BMS-345541, for 1 h. FOXC1 protein levels were examined by immunoblotting. (TIFF 56 kb

Additional file 1: Figure S1. of Identification of EGF-NF-κB-FOXC1 signaling axis in basal-like breast cancer

NF-κB transcription factor mediates EGF-induced FOXC1 expression in multiple breast cancer cell lines. a MDA-MB-231 and BT-20 cell lines were transiently co-transfected with the FOXC1 promoter-luc and NF-κB (p65), IκBα S32A/S36A super-repressor (p65 + SR-IκBα), or the vector. Reporter activities were assessed by luciferase assays. *, P < 0.05; **, P < 0.001. b MDA-MB-468 cells were transiently transfected with the IKKβ or SR-IκBα constructs and immunoblotted for FOXC1 expression. c MDA-MB-231 and BT-20 cell lines were treated with 100 ng/mL EGF for 2 h after preincubation with the NF-κB inhibitor Bay 11–7082 for 1 h. FOXC1 mRNA levels were examined using qRT-PCR. **, P < 0.001. d Total protein was extracted from MDA-MB-468, MDA-MB-231 and BT-20 cell lines after no starvation or treatment (Left to right, first three lanes) or after serum-starvation overnight with or without EGF treatment for 24 h. EGFR and phosphorylated EGFR (p-EGFR) levels were examined with immunoblotting. e BT-20 cells were treated with 100 ng/mL EGF for 1 h after preincubation with the Akt inhibitor-IV or U0126 (ERK inhibitor) for 45 min. Total protein was extracted and phospho-p65 (Ser 536) was examined by immunoblotting. (TIFF 131 kb