AMSP: Reducing Communication Overhead of ZeRO for Efficient LLM Training

Training large language models (LLMs) encounters challenges in GPU memory consumption due to the high memory requirements of model states. The widely used Zero Redundancy Optimizer (ZeRO) addresses this issue through strategic sharding but introduces communication challenges at scale. To tackle this problem, we propose AMSP, a system designed to optimize ZeRO for scalable LLM training. AMSP incorporates three flexible sharding strategies: Full-Replica, Full-Sharding, and Partial-Sharding, and allows each component within the model states (Parameters, Gradients, Optimizer States) to independently choose a sharding strategy as well as the device mesh. We conduct a thorough analysis of communication costs, formulating an optimization problem to discover the optimal sharding strategy. Additionally, AMSP optimizes distributed LLM training by efficiently overlapping communication with computation. Evaluations demonstrate up to 52\% Model FLOPs Utilization (MFU) when training the LLaMA-based model on 1024 GPUs, resulting in a 1.56 times improvement in training throughput compared to newly proposed systems like MiCS and ZeRO++

Mitigation of chronic unpredictable stress–induced cognitive deficits in mice by Lycium barbarum L (Solanaceae) polysaccharides

Purpose: To investigate the neuroprotective effects of Lycium barbarum polysaccharide (LBP) against concomitant cognitive dysfunction and changes in hippocampal CREB-BDNF signaling pathway in chronically unpredictable stressed mice.Methods: The mice were subjected to different unpredictable stressors for a period of 4 weeks. Behavioral tests, including open field (OFT) and Morris water maze (MWMT) tests were used to evaluate pharmacological effects. Serum corticosterone levels, protein expression level of BDNF and pCREB/CREB in hippocampus were assessed by ELISA, Western blot and immunohistochemistry methods, respectively. Morphological changes in pyramidal neurons in the hippocampus were studied by Nissl staining.Results: LBP improved mice performance in MWMT, indicating that it reversed chronic unpredictable stress (CUS)-induced cognitive deficits. LBP treatment reduced serum corticosterone levels and prevented neuron loss in the hippocampus. It maintained expression levels of BDNF and phosphorylation of CREB in hippocampus during CUS procedure.Conclusion: Lycium barbarum polysaccharide protects CREB-BDNF signaling pathway in hippocampus and relieves CUS-induced cognitive deficits. These results suggest that Lycium barbarum polysaccharides is potentially an alternative neuro-protective agent against stress-induced psychopathological dysfunction.Keywords: Lycium barbarum, Polysaccharide, Chronic unpredictable stress, Cognitive deficits, Brainderived neurotrophic factor, Calcium/cyclic-AMP responsive binding protei

Analysis of EGFR signaling pathway in nasopharyngeal carcinoma cells by quantitative phosphoproteomics

<p>Abstract</p> <p>Background</p> <p>The epidermal growth factor receptor (EGFR) is usually overexpressed in nasopharyngeal carcinoma (NPC) and is associated with pathogenesis of NPC. However, the downstream signaling proteins of EGFR in NPC have not yet been completely understood at the system level. The aim of this study was identify novel downstream proteins of EGFR signaling pathway in NPC cells.</p> <p>Results</p> <p>We analyzed EGFR-regulated phosphoproteome in NPC CNE2 cells using 2D-DIGE and mass spectrometry analysis after phosphoprotein enrichment. As a result, 33 nonredundant phosphoproteins including five known EGFR-regulated proteins and twenty-eight novel EGFR-regulated proteins in CNE2 were identified, three differential phosphoproteins were selectively validated, and two differential phosphoproteins (GSTP1 and GRB2) were showed interacted with phospho-EGFR. Bioinformatics analysis showed that 32 of 33 identified proteins contain phosphorylation modification sites, and 17 identified proteins are signaling proteins. GSTP1, one of the EGFR-regulated proteins, associated with chemoresistance was analyzed. The results showed that GSTP1 could contribute to paclitaxel resistance in EGF-stimulated CNE2 cells. Furthermore, an EGFR signaling network based on the identified EGFR-regulated phosphoproteins were constructed using Pathway Studio 5.0 software, which includes canonical and novel EGFR-regulated proteins and implicates the possible biological roles for those proteins.</p> <p>Conclusion</p> <p>The data not only can extend our knowledge of canonical EGFR signaling, but also will be useful to understand the molecular mechanisms of EGFR in NPC pathogenesis and search therapeutic targets for NPC.</p

Structural mechanism for bacterial oxidation of oceanic trimethylamine into trimethylamine N -oxide

Trimethylamine (TMA) and trimethylamine N-oxide (TMAO) are widespread in the ocean and are important nitrogen source for bacteria. TMA monooxygenase (Tmm), a bacterial flavin-containing monooxygenase (FMO), is found widespread in marine bacteria and is responsible for converting TMA to TMAO. However, the molecular mechanism of TMA oxygenation by Tmm has not been explained. Here, we determined the crystal structures of two reaction intermediates of a marine bacterial Tmm (RnTmm) and elucidated the catalytic mechanism of TMA oxidation by RnTmm. The catalytic process of Tmm consists of a reductive half-reaction and an oxidative half-reaction. In the reductive half-reaction, FAD is reduced and a C4a-hydroperoxyflavin intermediate forms. In the oxidative half-reaction, this intermediate attracts TMA through electronic interactions. After TMA binding, NADP+ bends and interacts with D317, shutting off the entrance to create a protected micro-environment for catalysis and exposing C4a-hydroperoxyflavin to TMA for oxidation. Sequence analysis suggests that the proposed catalytic mechanism is common for bacterial Tmms. These findings reveal the catalytic process of TMA oxidation by marine bacterial Tmm and first show that NADP+ undergoes a conformational change in the oxidative half-reaction of FMOs

A role for the cerebellum in motor-triggered alleviation of anxiety

Physical exercise is known to reduce anxiety, but the underlying brain mechanisms remain unclear. Here, we explore a hypothalamo-cerebello-amygdalar circuit that may mediate motor-dependent alleviation of anxiety. This three-neuron loop, in which the cerebellar dentate nucleus takes center stage, bridges the motor system with the emotional system. Subjecting animals to a constant rotarod engages glutamatergic cerebellar dentate neurons that drive PKCδ+ amygdalar neurons to elicit an anxiolytic effect. Moreover, challenging animals on an accelerated rather than a constant rotarod engages hypothalamic neurons that provide a superimposed anxiolytic effect via an orexinergic projection to the dentate neurons that activate the amygdala. Our findings reveal a cerebello-limbic pathway that may contribute to motor-triggered alleviation of anxiety and that may be optimally exploited during challenging physical exercise.</p

Rapid Surface Oxidation as a Source of Surface Degradation Factor for Bi2Se3

Bi2Se3 is a topological insulator with metallic surface states residing in a large bulk bandgap. It is believed that Bi2Se3 gets additional n-type doping after exposure to atmosphere, thereby reducing the relative contribution of surface states in total conductivity. In this letter, transport measurements on Bi2Se3 nanoribbons provide additional evidence of such environmental doping process. Systematic surface composition analyses by X-ray photoelectron spectroscopy reveal fast formation and continuous growth of native oxide on Bi2Se3 under ambient conditions. In addition to n-type doping at the surface, such surface oxidation is likely the material origin of the degradation of topological surface states. Appropriate surface passivation or encapsulation may be required to probe topological surface states of Bi2Se3 by transport measurements

Chemical profiling of Sanjin tablets and exploration of their effective substances and mechanism in the treatment of urinary tract infections

Introduction: Sanjin tablets (SJT) are a well-known Chinese patent drug that have been used to treat urinary tract infections (UTIs) for the last 40 years. The drug consists of five herbs, but only 32 compounds have been identified, which hinders the clarification of its effective substances and mechanism.Methods: The chemical constituents of SJT and their effective substances and functional mechanism involved in the treatment of UTIs were investigated by using high performance liquid chromatography-electrospray ionization-ion trap-time of flight-mass spectrometry (HPLC-ESI-IT-TOF-MSn), network pharmacology, and molecular docking.Results: A total of 196 compounds of SJT (SJT-MS) were identified, and 44 of them were unequivocally identified by comparison with the reference compounds. Among 196 compounds, 13 were potential new compounds and 183 were known compounds. Among the 183 known compounds, 169 were newly discovered constituents of SJT, and 93 compounds were not reported in the five constituent herbs. Through the network pharmacology method, 119 targets related to UTIs of 183 known compounds were predicted, and 20 core targets were screened out. Based on the “compound–target” relationship analysis, 94 compounds were found to act on the 20 core targets and were therefore regarded as potential effective compounds. According to the literature, 27 of the 183 known compounds were found to possess antimicrobial and anti-inflammatory activities and were verified as effective substances, of which 20 were first discovered in SJT. Twelve of the 27 effective substances overlapped with the 94 potential effective compounds and were determined as key effective substances of SJT. The molecular docking results showed that the 12 key effective substances and 10 selected targets of the core targets have good affinity for each other.Discussion: These results provide a solid foundation for understanding the effective substances and mechanism of SJT

Synchronous post-acceleration of laser-driven protons in helical coil targets by controlling the current dispersion

Post-acceleration of protons in helical coil targets driven by intense, ultrashort laser pulses can enhance ion energy by utilizing the transient current from the targets’ self-discharge. The acceleration length of protons can exceed a few millimeters, and the acceleration gradient is of the order of GeV/m. How to ensure the synchronization between the accelerating electric field and the protons is a crucial problem for efficient post-acceleration. In this paper, we study how the electric field mismatch induced by current dispersion affects the synchronous acceleration of protons. We propose a scheme using a two-stage helical coil to control the current dispersion. With optimized parameters, the energy gain of protons is increased by four times. Proton energy is expected to reach 45 MeV using a hundreds-of-terawatts laser, or more than 100 MeV using a petawatt laser, by controlling the current dispersion

The gut microbiome in atherosclerotic cardiovascular disease

The gut microbiota may play a role in cardiovascular diseases. Here, the authors perform a metagenome-wide association study on stools from individuals with atherosclerotic cardiovascular disease and healthy controls, identifying microbial strains and functions associated with the disease