Elixir: Train a Large Language Model on a Small GPU Cluster

In recent years, the number of parameters of one deep learning (DL) model has been growing much faster than the growth of GPU memory space. People who are inaccessible to a large number of GPUs resort to heterogeneous training systems for storing model parameters in CPU memory. Existing heterogeneous systems are based on parallelization plans in the scope of the whole model. They apply a consistent parallel training method for all the operators in the computation. Therefore, engineers need to pay a huge effort to incorporate a new type of model parallelism and patch its compatibility with other parallelisms. For example, Mixture-of-Experts (MoE) is still incompatible with ZeRO-3 in Deepspeed. Also, current systems face efficiency problems on small scale, since they are designed and tuned for large-scale training. In this paper, we propose Elixir, a new parallel heterogeneous training system, which is designed for efficiency and flexibility. Elixir utilizes memory resources and computing resources of both GPU and CPU. For flexibility, Elixir generates parallelization plans in the granularity of operators. Any new type of model parallelism can be incorporated by assigning a parallel pattern to the operator. For efficiency, Elixir implements a hierarchical distributed memory management scheme to accelerate inter-GPU communications and CPU-GPU data transmissions. As a result, Elixir can train a 30B OPT model on an A100 with 40GB CUDA memory, meanwhile reaching 84% efficiency of Pytorch GPU training. With its super-linear scalability, the training efficiency becomes the same as Pytorch GPU training on multiple GPUs. Also, large MoE models can be trained 5.3x faster than dense models of the same size. Now Elixir is integrated into ColossalAI and is available on its main branch

Efficacy and safety of perampanel in patients with drug-resistant partial seizures after conversion from double-blind placebo to open-label perampanel

SummaryObjectiveTo evaluate the efficacy and safety of perampanel in patients with drug-resistant partial seizures after the conversion from double-blind placebo in three phase III studies to open-label perampanel, and to assess the impact of perampanel titration rates through a comparison of weekly vs biweekly dose increases.MethodsPatients who completed the three multinational, double-blind, placebo-controlled, phase III core studies (studies 304, 305, or 306) were eligible to enroll in the extension study (study 307). Patients completing the double-blind treatment (6-week titration, 13-week maintenance) with placebo (DB-PBO) or perampanel (DB-PER) began the extension study with a 16-week blinded conversion period, during which DB-PBO patients were switched to perampanel. Doses were titrated in 2-mg increments (biweekly) to an individualized maximum tolerated dose of perampanel (up to 12mg/day). Patients then entered a planned, open-label treatment period.ResultsPerampanel treatment during the extension study reduced total seizure frequency/28 days relative to the double-blind prerandomization baseline regardless of prior perampanel or placebo treatment in the core studies. In the DB-PBO patients, median percent reductions in seizure frequency at the end of the double-blind period, at the end of the conversion period, and at Weeks 40–52 in the open-label maintenance period were 18.6%, 44.3%, and 55.0%, respectively. Seizure control was also improved in the DB-PER patients during the extension period compared to the end of the double-blind period. Responder rates were similar between the 2 patient groups at the end of the conversion period. Perampanel was well tolerated, with the most common treatment-emergent adverse events being dizziness, somnolence, weight increase, irritability, fatigue, and headache. For those patients randomized to the 12mg group (DB-PER 12mg), 78.4% reached the daily dose of 10 or 12mg by the end of the 6-week titration period of the double-blind phase. By the end of the 16-week conversion period of the extension study, 64.0% of DB-PBO patients reached the daily dose of 10 or 12mg. Seizure frequency reduction was greater after the first 13-week maintenance period of the extension study in the DB-PBO group compared to patients assigned to DB-PER 12mg during the 13-week maintenance period of the double-blind study.ConclusionPatients who received placebo in the phase III core DB studies and transitioned to perampanel in the open-label extension study (DB-PBO) achieved seizure control at the end of the conversion period similar to that of patients who had been previously exposed to perampanel (DB-PER) as well as comparable safety outcomes. Patients who received perampanel during the core studies and continued with treatment during the extension study (DB-PER) also showed sustained improvements in seizure control with long-term exposure to perampanel

Human germline and pan-cancer variomes and their distinct functional profiles

Identification of non-synonymous single nucleotide variations (nsSNVs) has exponentially increased due to advances in Next-Generation Sequencing technologies. The functional impacts of these variations have been difficult to ascertain because the corresponding knowledge about sequence functional sites is quite fragmented. It is clear that mapping of variations to sequence functional features can help us better understand the pathophysiological role of variations. In this study, we investigated the effect of nsSNVs on more than 17 common types of post-translational modification (PTM) sites, active sites and binding sites. Out of 1 705 285 distinct nsSNVs on 259 216 functional sites we identified 38 549 variations that significantly affect 10 major functional sites. Furthermore, we found distinct patterns of site disruptions due to germline and somatic nsSNVs. Pan-cancer analysis across 12 different cancer types led to the identification of 51 genes with 106 nsSNV affected functional sites found in 3 or more cancer types. 13 of the 51 genes overlap with previously identified Significantly Mutated Genes (Nature. 2013 Oct 17;502(7471)). 62 mutations in these 13 genes affecting functional sites such as DNA, ATP binding and various PTM sites occur across several cancers and can be prioritized for additional validation and investigations

BioXpress: an integrated RNA-seq-derived gene expression database for pan-cancer analysis.

BioXpress is a gene expression and cancer association database in which the expression levels are mapped to genes using RNA-seq data obtained from The Cancer Genome Atlas, International Cancer Genome Consortium, Expression Atlas and publications. The BioXpress database includes expression data from 64 cancer types, 6361 patients and 17 469 genes with 9513 of the genes displaying differential expression between tumor and normal samples. In addition to data directly retrieved from RNA-seq data repositories, manual biocuration of publications supplements the available cancer association annotations in the database. All cancer types are mapped to Disease Ontology terms to facilitate a uniform pan-cancer analysis. The BioXpress database is easily searched using HUGO Gene Nomenclature Committee gene symbol, UniProtKB/RefSeq accession or, alternatively, can be queried by cancer type with specified significance filters. This interface along with availability of pre-computed downloadable files containing differentially expressed genes in multiple cancers enables straightforward retrieval and display of a broad set of cancer-related genes

final safety tolerability and seizure outcomes in patients with focal epilepsy treated with adjunctive perampanel for up to 4 years in an open label extension of phase iii randomized trials study 307

n/

Time-course adaptive changes in hippocampal transcriptome and synaptic function induced by simulated microgravity associated with cognition

IntroductionThe investigation of cognitive function in microgravity, both short-term and long-term, remains largely descriptive. And the underlying mechanisms of the changes over time remain unclear.MethodsBehavioral tests, electrophysiological recording, and RNA sequencing were used to observe differences in behavior, synaptic plasticity, and gene expression.ResultsInitially, we measured the performance of spatial cognition exposed to long-term simulated microgravity (SM). Both working memory and advanced cognitive abilities were enhanced. Somewhat surprisingly, the synaptic plasticity of the hippocampal CA3-CA1 synapse was impaired. To gain insight into the mechanism of changing regularity over time, transcriptome sequencing in the hippocampus was performed. The analysis identified 20 differentially expressed genes (DEGs) in the hippocampus after short-term modeling, 19 of which were up-regulated. Gene Ontology (GO) analysis showed that these up-regulated genes were mainly enriched in synaptic-related processes, such as Stxbp5l and Epha6. This might be related to the enhancement of working memory performance under short-term SM exposure. Under exposure to long-term SM, 7 DEGs were identified in the hippocampus, all of which were up-regulated and related to oxidative stress and metabolism, such as Depp1 and Lrg1. Compensatory effects occurred with increased modeling time.DiscussionTo sum up, our current research indicates that the cognitive function under SM exposure is consistently maintained or potentially even being enhanced over both short and long durations. The underlying mechanisms are intricate and potentially linked to the differential expression of hippocampal-associated genes and alterations in synaptic function, with these effects being time-dependent. The present study will lay the experimental and theoretical foundation of the multi-level mechanism of cognitive function under space flight

Multivalent HA DNA Vaccination Protects against Highly Pathogenic H5N1 Avian Influenza Infection in Chickens and Mice

Sustained outbreaks of highly pathogenic avian influenza (HPAI) H5N1 in avian species increase the risk of reassortment and adaptation to humans. The ability to contain its spread in chickens would reduce this threat and help maintain the capacity for egg-based vaccine production. While vaccines offer the potential to control avian disease, a major concern of current vaccines is their potency and inability to protect against evolving avian influenza viruses.The ability of DNA vaccines encoding hemagglutinin (HA) proteins from different HPAI H5N1 serotypes was evaluated for its ability to elicit neutralizing antibodies and to protect against homologous and heterologous HPAI H5N1 strain challenge in mice and chickens after DNA immunization by needle and syringe or with a pressure injection device. These vaccines elicited antibodies that neutralized multiple strains of HPAI H5N1 when given in combinations containing up to 10 HAs. The response was dose-dependent, and breadth was determined by the choice of the influenza virus HA in the vaccine. Monovalent and trivalent HA vaccines were tested first in mice and conferred protection against lethal H5N1 A/Vietnam/1203/2004 challenge 68 weeks after vaccination. In chickens, protection was observed against heterologous strains of HPAI H5N1 after vaccination with a trivalent H5 serotype DNA vaccine with doses as low as 5 microg DNA given twice either by intramuscular needle injection or with a needle-free device.DNA vaccines offer a generic approach to influenza virus immunization applicable to multiple animal species. In addition, the ability to substitute plasmids encoding different strains enables rapid adaptation of the vaccine to newly evolving field isolates

Search for dark matter produced in association with bottom or top quarks in √s = 13 TeV pp collisions with the ATLAS detector

A search for weakly interacting massive particle dark matter produced in association with bottom or top quarks is presented. Final states containing third-generation quarks and miss- ing transverse momentum are considered. The analysis uses 36.1 fb−1 of proton–proton collision data recorded by the ATLAS experiment at √s = 13 TeV in 2015 and 2016. No significant excess of events above the estimated backgrounds is observed. The results are in- terpreted in the framework of simplified models of spin-0 dark-matter mediators. For colour- neutral spin-0 mediators produced in association with top quarks and decaying into a pair of dark-matter particles, mediator masses below 50 GeV are excluded assuming a dark-matter candidate mass of 1 GeV and unitary couplings. For scalar and pseudoscalar mediators produced in association with bottom quarks, the search sets limits on the production cross- section of 300 times the predicted rate for mediators with masses between 10 and 50 GeV and assuming a dark-matter mass of 1 GeV and unitary coupling. Constraints on colour- charged scalar simplified models are also presented. Assuming a dark-matter particle mass of 35 GeV, mediator particles with mass below 1.1 TeV are excluded for couplings yielding a dark-matter relic density consistent with measurements