TRANSOM: An Efficient Fault-Tolerant System for Training LLMs

Large language models (LLMs) with hundreds of billions or trillions of parameters, represented by chatGPT, have achieved profound impact on various fields. However, training LLMs with super-large-scale parameters requires large high-performance GPU clusters and long training periods lasting for months. Due to the inevitable hardware and software failures in large-scale clusters, maintaining uninterrupted and long-duration training is extremely challenging. As a result, A substantial amount of training time is devoted to task checkpoint saving and loading, task rescheduling and restart, and task manual anomaly checks, which greatly harms the overall training efficiency. To address these issues, we propose TRANSOM, a novel fault-tolerant LLM training system. In this work, we design three key subsystems: the training pipeline automatic fault tolerance and recovery mechanism named Transom Operator and Launcher (TOL), the training task multi-dimensional metric automatic anomaly detection system named Transom Eagle Eye (TEE), and the training checkpoint asynchronous access automatic fault tolerance and recovery technology named Transom Checkpoint Engine (TCE). Here, TOL manages the lifecycle of training tasks, while TEE is responsible for task monitoring and anomaly reporting. TEE detects training anomalies and reports them to TOL, who automatically enters the fault tolerance strategy to eliminate abnormal nodes and restart the training task. And the asynchronous checkpoint saving and loading functionality provided by TCE greatly shorten the fault tolerance overhead. The experimental results indicate that TRANSOM significantly enhances the efficiency of large-scale LLM training on clusters. Specifically, the pre-training time for GPT3-175B has been reduced by 28%, while checkpoint saving and loading performance have improved by a factor of 20.Comment: 14 pages, 9 figure

Theoretical foundations of studying criticality in the brain

Criticality is hypothesized as a physical mechanism underlying efficient transitions between cortical states and remarkable information processing capacities in the brain. While considerable evidence generally supports this hypothesis, non-negligible controversies persist regarding the ubiquity of criticality in neural dynamics and its role in information processing. Validity issues frequently arise during identifying potential brain criticality from empirical data. Moreover, the functional benefits implied by brain criticality are frequently misconceived or unduly generalized. These problems stem from the non-triviality and immaturity of the physical theories that analytically derive brain criticality and the statistic techniques that estimate brain criticality from empirical data. To help solve these problems, we present a systematic review and reformulate the foundations of studying brain criticality, i.e., ordinary criticality (OC), quasi-criticality (qC), self-organized criticality (SOC), and self-organized quasi-criticality (SOqC), using the terminology of neuroscience. We offer accessible explanations of the physical theories and statistic techniques of brain criticality, providing step-by-step derivations to characterize neural dynamics as a physical system with avalanches. We summarize error-prone details and existing limitations in brain criticality analysis and suggest possible solutions. Moreover, we present a forward-looking perspective on how optimizing the foundations of studying brain criticality can deepen our understanding of various neuroscience questions

Time to Clinical Benefit of Intensive Blood Pressure Lowering in Patients 60 Years and Older With Hypertension

Importance Recent guidelines recommend a systolic blood pressure (BP) goal of less than 150 mm Hg or even 130 mm Hg for adults aged 60 years or older. However, harms from intensive BP treatments occur immediately (eg, syncope, fall), and benefits for cardiovascular event reduction emerge over time. Therefore, harms with low chance of benefit need to be clearer, particularly for those with limited life expectancy. Objective To estimate the time needed to potentially derive clinical benefit from intensive BP treatment in patients 60 years and older. Design, Setting, and Participants This secondary analysis included individual patient data from published randomized clinical trials with 27 414 patients 60 years or older with hypertension. Patient-level survival data were reconstructed when the original data were not available. Published trials were identified by searching PubMed until October 15, 2021. Exposures Intensive BP lowering vs standard BP lowering with the treat-to-target design. Main Outcomes and Measures Major adverse cardiovascular event (MACE) defined by each trial, which was broadly similar with all trials including myocardial infarction, stroke, and cardiovascular mortality. Results Six trials (original data from 2 trials and reconstructed data from 4 trials) with 27 414 participants (mean age, 70 years; 56.3% were women) were included in the analysis. Intensive BP treatment with a systolic BP target below 140 mm Hg was significantly associated with a 21% reduction in MACE (hazard ratio, 0.79; 95% CI, 0.71-0.88; P < .001). On average, 9.1 (95% CI, 4.0-20.6) months were needed to prevent 1 MACE per 500 patients with the intensive BP treatment (absolute risk reduction [ARR], 0.002). Likewise, 19.1 (95% CI, 10.9-34.2) and 34.4 (95% CI, 22.7-59.8) months were estimated to avoid 1 MACE per 200 (ARR, 0.005) and 100 (ARR, 0.01) patients, respectively. Conclusions and Relevance In this analysis, findings suggest that for patients 60 years and older with hypertension, intensive BP treatment may be appropriate for some adults with a life expectancy of greater than 3 years but may not be suitable for those with less than 1 year

Time to Benefit of Sodium-Glucose Cotransporter-2 Inhibitors Among Patients With Heart Failure

IMPORTANCE Emerging evidence has consistently demonstrated that sodium-glucose cotransporter 2 (SGLT2) inhibitors reduce the risk of heart failure (HF) hospitalization and cardiovascular (CV) death among patients with HF. However, it remains unclear how long a patient needs to live to potentially benefit from SGLT2 inhibitors in this population. OBJECTIVES To estimate the time to benefit from SGLT2 inhibitors among patients with HF. DESIGN, SETTING, AND PARTICIPANTS This comparative effectiveness study systematically searched PubMed for completed randomized clinical trials about SGLT2 inhibitors and patients with HF published until September 5, 2022; 5 trials with the year of publication ranging from 2019 to 2022 were eventually included. Statistical analysis was performed from April to October 2022. INTERVENTION Addition of SGLT2 inhibitors or placebo to guideline-recommended therapy. MAIN OUTCOMES AND MEASURES The primary outcome was the time to first event of CV death or worsening HF, which was broadly comparable across the included trials. RESULTS Five trials consisting of 21 947 patients with HF (7837 [35.7%] were female; mean or median age older than 65 years within each trial) were included. SGLT2 inhibitors significantly reduced the risk of worsening HF or CV death (hazard ratio [HR], 0.77 [95% CI, 0.73-0.82]). Time to first nominal statistical significance (P < .05) was 26 days (0.86 months), and statistical significance was sustained from day 118 (3.93 months) onwards. A mean of 0.19 (95% CI, 0.12-0.35) months were needed to prevent 1 worsening HF or CV death per 500 patients with SGLT2 inhibitors (absolute risk reduction [ARR], 0.002). Likewise, 0.66 (95% CI, 0.43-1.13) months was estimated to avoid 1 event per 200 patients with SGLT2 inhibitors (ARR, 0.005), 1.74 (95% CI, 1.07-2.61) months to avoid 1 event per 100 patients (ARR, 0.010), and 4.96 (95% CI, 3.18-7.26) months to avoid 1 event per 50 patients (ARR, 0.020). Further analyses indicated a shorter time to benefit for HF hospitalization and among patients with diabetes or HF with reduced ejection fraction. CONCLUSIONS AND RELEVANCE In this comparative effectiveness research study of estimating the time to benefit from SGLT2 inhibitors among patients with HF, a rapid clinical benefit in reducing CV death or worsening HF was found, suggesting that their use may be beneficial for most individuals with HF

Time‐weighted blood pressure with cardiovascular risk among patients with or without diabetes

Background: Usual measures of blood pressure (BP) do not account for both the magnitude and duration of exposure to elevated BP over time. We aimed to demonstrate the effect of a novel time‐weighted BP on cardiovascular outcomes using a post hoc analysis of two published randomized trials. Hypothesis: Time‐weighted blood pressure is associated with cardiovascular risk among patients with or without diabetes. Methods: The limited‐access ACCORD and SPRINT data sets were used for the current study. Time‐weighted BP is obtained by dividing cumulative BP by the total follow‐up time. Time‐weighted BP burden above a threshold is also determined after deriving the time‐weighted BP by re‐zeroing the interpolated pressure values at two different hypertension thresholds (>140/90 and >130/80 mmHg). Results: Eighteen thousand five hundred forty‐one patients from the two clinical trials were enrolled in this study. A J‐curve relation was observed between time‐weighted BP and major cardiovascular events (MACE). The systolic blood pressure (SBP) burden independently predicted MACE across the two trials at different thresholds (ACCORD: SBP > 130 mmHg, HR = 1.05 [1.03−1.06]; SBP > 140 mmHg, HR = 1.06 [1.04−1.08]; SPRINT: SBP > 130 mmHg, HR = 1.04 [1.03−1.05]; SBP > 140 mmHg, HR = 1.05 [1.04−1.07]). Consistent results were found for diastolic blood pressure (DBP) burden (ACCORD: DBP > 80 mmHg, HR = 1.10 [1.06−1.15]; DBP > 90 mmHg, HR = 1.20 [1.11−1.30]. SPRINT: DBP > 80 mmHg, HR = 1.06 [1.02−1.09]; DBP > 90 mmHg, HR = 1.12 [1.06−1.18]). Significant associations were also observed for stroke, myocardial infarction, cardiovascular death, and all‐cause mortality. Conclusion: Both time‐weighted SBP and DBP independently influenced the risk of adverse cardiovascular events among patients with and without diabetes, regardless of the definition of hypertension (130/80 or <140/90 mmHg)

Plasma exosomal miR-320d, miR-4479, and miR-6763-5p as diagnostic biomarkers in epithelial ovarian cancer

BackgroundExosomal miRNA had been proved as the promising biomarkers for multiple cancers including epithelial ovarian cancer (EOC). This study aimed to validate the diagnostic accuracy of exosomal miR-320d, miR-4479, and miR-6763-5p for EOC.Materials and methodsExosomes isolated from the plasma by ultracentrifugation were verified using TEM, qNano and western blot. MiRNAs sequencing was used to screen out the differential exosomal miRNAs and miR-320d, miR-4479, and miR-6763-5p were selected as candidates, which were further verified by RT-qPCR in 168 healthy donors and 161 primary EOC patients. Besides, the diagnostic accuracy of these three exosomal miRNAs were evaluated using the receiver operating characteristic curve (ROC).ResultsMiRNAs sequencing revealed 95 differential exosomal miRNAs between EOC patients and healthy donors. Subsequently, exosomal miR-320d, miR-4479, and miR-6763-5p were significantly down regulated in EOC patients compared with healthy controls and benign patients. More importantly, these three miRNAs could serve as circulating diagnostics biomarkers for EOC, possessing areas under the curve (AUC) of 0.6549, 0.7781, and 0.6834, respectively. Moreover, these three exosomal miRNAs levels were closely associated with lymph node metastasis, meanwhile exosomal miR-320d and miR-4479 expression was related to tumor stage.ConclusionExosomal miR-320d, miR-4479, and miR-6763-5p might serve as potential biomarkers for EOC

Critical Dynamic Stress and Cumulative Plastic Deformation of Calcareous Sand Filler Based on Shakedown Theory

Calcareous sand is a special marine soil rich in calcium carbonate minerals, characterized by brittle particles. It is, therefore, widely used as a filling material in the construction of islands and reefs. In this study, a series of cyclic tri-axial tests were conducted on calcareous sand taken from a reef in the South China Sea under different confining pressures and cyclic stress ratio (CSR). Then, applying the shakedown theory, the cumulative deformation of calcareous sand under a long-term cyclic load of aircraft was evaluated. Results showed that with the increase in the effective confining pressure, the stress–strain curves of calcareous sand showed a change from the strain-softening to the strain-hardening state; the volumetric strain of calcareous sand showed a change from shear shrinkage and then shear expansion to continuous shear shrinkage. Calcareous sand showed three different response behaviors under cyclic load: plastic shakedown, plastic creep and incremental plastic failure. With the plastic strain rate as the defining index, this study determined the critical CSR of calcareous sand under different shakedown response statuses and found them to increase with the effective confining pressure. The empirical formula for critical stress was established based on the fitting analysis of critical CSR under different confining pressures, taking the confining pressure as the variable. At the early stage of the cyclic load, calcareous sand samples were under compression. When the resilient modulus grew rapidly and the number of loading cycles continued to increase, the particles of calcareous sand samples were crushed, causing the fine particles to fill the voids among coarse particles, further compacting the samples and increasing the resilient modulus of calcareous sand samples. Hardin’s breakage potential model was adopted to quantitatively describe the particle breakage of calcareous sand samples before and after tests. The results indicated that calcareous sand samples produced obvious particle breakage when the CSR was small. As the CSR increased, the extent of the breakage of the sample particles first increased and thereafter stabilized. This study provides a theoretical reference for the assessment of the dynamic stability of calcareous sand subgrade subjected to traffic loads