Beyond neural scaling laws: beating power law scaling via data pruning

Widely observed neural scaling laws, in which error falls off as a power of the training set size, model size, or both, have driven substantial performance improvements in deep learning. However, these improvements through scaling alone require considerable costs in compute and energy. Here we focus on the scaling of error with dataset size and show how in theory we can break beyond power law scaling and potentially even reduce it to exponential scaling instead if we have access to a high-quality data pruning metric that ranks the order in which training examples should be discarded to achieve any pruned dataset size. We then test this improved scaling prediction with pruned dataset size empirically, and indeed observe better than power law scaling in practice on ResNets trained on CIFAR-10, SVHN, and ImageNet. Next, given the importance of finding high-quality pruning metrics, we perform the first large-scale benchmarking study of ten different data pruning metrics on ImageNet. We find most existing high performing metrics scale poorly to ImageNet, while the best are computationally intensive and require labels for every image. We therefore developed a new simple, cheap and scalable self-supervised pruning metric that demonstrates comparable performance to the best supervised metrics. Overall, our work suggests that the discovery of good data-pruning metrics may provide a viable path forward to substantially improved neural scaling laws, thereby reducing the resource costs of modern deep learning.Comment: Outstanding Paper Award @ NeurIPS 2022. Added github link to metric score

The unfolded protein response affects readthrough of premature termination codons

One-third of monogenic inherited diseases result from premature termination codons (PTCs). Readthrough of in-frame PTCs enables synthesis of full-length functional proteins. However, extended variability in the response to readthrough treatment is found among patients, which correlates with the level of nonsense transcripts. Here, we aimed to reveal cellular pathways affecting this inter-patient variability. We show that activation of the unfolded protein response (UPR) governs the response to readthrough treatment by regulating the levels of transcripts carrying PTCs. Quantitative proteomic analyses showed substantial differences in UPR activation between patients carrying PTCs, correlating with their response. We further found a significant inverse correlation between the UPR and nonsense-mediated mRNA decay (NMD), suggesting a feedback loop between these homeostatic pathways. We uncovered and characterized the mechanism underlying this NMD-UPR feedback loop, which augments both UPR activation and NMD attenuation. Importantly, this feedback loop enhances the response to readthrough treatment, highlighting its clinical importance. Altogether, our study demonstrates the importance of the UPR and its regulatory network for genetic diseases caused by PTCs and for cell homeostasis under normal conditions

Identification of a tripartite basal promoter which regulates human terminal deoxynucleotidyl transferase gene expression.

In order to locate the promoter region of the human terminal deoxynucleotidyl transferase gene, serially truncated segments of the 5'-flanking region of the gene were cloned into a chloramphenicol acetyltransferase reporter vector. Transient transfection analyses of the terminal transferase-reporter gene constructs identified the basal promoter region within -34 to +40 base pairs relative to the transcription start site. Three promoter elements were defined in this region. The primary element is within 34 base pairs upstream of the transcription start site. The CAP site is 62 base pairs upstream of the translation start site. The secondary element involves sequences around the transcription start site. The third is located 25 base pairs downstream from the initiation site (+25 to +40). This tripartite basal promoter was not tissue specific; similar patterns of promoter activity were observed in terminal transferase expressing and non-expressing cells. Transfection analyses also indicated the presence of negative regulatory elements upstream of the basal promoter region, and these elements were preferentially active in cells expressing terminal transferase

Nucleotide pool imbalance and adenosine deaminase deficiency induce alterations of N-region insertions during V(D)J recombination

Template-independent nucleotide additions (N regions) generated at sites of V(D)J recombination by terminal deoxynucleotidyl transferase (TdT) increase the diversity of antigen receptors. Two inborn errors of purine metabolism, deficiencies of adenosine deaminase (ADA) and purine nucleoside phosphorylase (PNP), result in defective lymphoid development and aberrant pools of 2′-deoxynucleotides that are substrates for TdT in lymphoid precursors. We have asked whether selective increases in dATP or dGTP pools result in altered N regions in an extrachromosomal substrate transfected into T-cell or pre–B-cell lines. Exposure of the transfected cells to 2′-deoxyadenosine and an ADA inhibitor increased the dATP pool and resulted in a marked increase in A–T insertions at recombination junctions, with an overall decreased frequency of V(D)J recombination. Sequence analysis of VH-DH-JH junctions from the IgM locus in B-cell lines from ADA-deficient patients demonstrated an increase in A–T insertions equivalent to that found in the transfected cells. In contrast, elevation of dGTP pools, as would occur in PNP deficiency, did not alter the already rich G–C content of N regions. We conclude that the frequency of V(D)J recombination and the composition of N-insertions are influenced by increases in dATP levels, potentially leading to alterations in antigen receptors and aberrant lymphoid development. Alterations in N-region insertions may contribute to the B-cell dysfunction associated with ADA deficiency

Effect of expression of adenine phosphoribosyltransferase on the in vivo anti-tumor activity of prodrugs activated by E. coli purine nucleoside phosphorylase

The use of E. coli purine nucleoside phosphorylase (PNP) to activate prodrugs has demonstrated excellent activity in the treatment of various human tumor xenografts in mice. E. coli PNP cleaves purine nucleoside analogs to generate toxic adenine analogs, which are activated by adenine phosphoribosyl transferase (APRT) to metabolites that inhibit RNA and protein synthesis. We created tumor cell lines that encode both E. coli PNP and excess levels of human APRT, and have used these new cell models to test the hypothesis that treatment of otherwise refractory human tumors could be enhanced by overexpression of APRT. In vivo studies with 6-methylpurine-2′-deoxyriboside (MeP-dR), 2-F-2′-deoxyadenosine (F-dAdo) or 9-β-D-arabinofuranosyl-2-fluoroadenine 5′-monophosphate (F-araAMP) indicated that increased APRT in human tumor cells coexpressing E. coli PNP did not enhance either the activation or the anti-tumor activity of any of the three prodrugs. Interestingly, expression of excess APRT in bystander cells improved the activity of MeP-dR, but diminished the activity of F-araAMP. In vitro studies indicated that increasing the expression of APRT in the cells did not significantly increase the activation of MeP. These results provide insight into the mechanism of bystander killing of the E. coli PNP strategy, and suggest ways to enhance the approach that are independent of APRT

From CFTR biology toward combinatorial pharmacotherapy:expanded classification of cystic fibrosis mutations

More than 2000 mutations in the cystic fibrosis transmembrane conductance regulator (CFTR) have been described that confer a range of molecular cell biological and functional phenotypes. Most of these mutations lead to compromised anion conductance at the apical plasma membrane of secretory epithelia and cause cystic fibrosis (CF) with variable disease severity. Based on the molecular phenotypic complexity of CFTR mutants and their susceptibility to pharmacotherapy, it has been recognized that mutations may impose combinatorial defects in CFTR channel biology. This notion led to the conclusion that the combination of pharmacotherapies addressing single defects (e.g., transcription, translation, folding, and/or gating) may show improved clinical benefit over available low-efficacy monotherapies. Indeed, recent phase 3 clinical trials combining ivacaftor (a gating potentiator) and lumacaftor (a folding corrector) have proven efficacious in CF patients harboring the most common mutation (deletion of residue F508, ΔF508, or Phe508del). This drug combination was recently approved by the U.S. Food and Drug Administration for patients homozygous for ΔF508. Emerging studies of the structural, cell biological, and functional defects caused by rare mutations provide a new framework that reveals a mixture of deficiencies in different CFTR alleles. Establishment of a set of combinatorial categories of the previously defined basic defects in CF alleles will aid the design of even more efficacious therapeutic interventions for CF patients

Highly Efficient Gene Editing of Cystic Fibrosis Patient-Derived Airway Basal Cells Results in Functional CFTR Correction

There is a strong rationale to consider future cell therapeutic approaches for cystic fibrosis (CF) in which autologous proximal airway basal stem cells, corrected for CFTR mutations, are transplanted into the patient's lungs. We assessed the possibility of editing the CFTR locus in these cells using zinc-finger nucleases and have pursued two approaches. The first, mutation-specific correction, is a footprint-free method replacing the CFTR mutation with corrected sequences. We have applied this approach for correction of ΔF508, demonstrating restoration of mature CFTR protein and function in air-liquid interface cultures established from bulk edited basal cells. The second is targeting integration of a partial CFTR cDNA within an intron of the endogenous CFTR gene, providing correction for all CFTR mutations downstream of the integration and exploiting the native CFTR promoter and chromatin architecture for physiologically relevant expression. Without selection, we observed highly efficient, site-specific targeted integration in basal cells carrying various CFTR mutations and demonstrated restored CFTR function at therapeutically relevant levels. Significantly, Omni-ATAC-seq analysis revealed minimal impact on the positions of open chromatin within the native CFTR locus. These results demonstrate efficient functional correction of CFTR and provide a platform for further ex vivo and in vivo editing. © 2020 The American Society of Gene and Cell TherapySuzuki et al. report correction of the CFTR defect in cystic fibrosis airway basal stem cells. They utilized gene-editing strategies either specific for the ΔF508 CFTR mutation or applicable to most CFTR mutations. Both approaches yielded highly efficient correction without selection, restoring CFTR function to therapeutically relevant levels