Unsupervised learning on spontaneous retinal activity leads to efficient neural representation geometry

Prior to the onset of vision, neurons in the developing mammalian retina spontaneously fire in correlated activity patterns known as retinal waves. Experimental evidence suggests that retinal waves strongly influence the emergence of sensory representations before visual experience. We aim to model this early stage of functional development by using movies of neurally active developing retinas as pre-training data for neural networks. Specifically, we pre-train a ResNet-18 with an unsupervised contrastive learning objective (SimCLR) on both simulated and experimentally-obtained movies of retinal waves, then evaluate its performance on image classification tasks. We find that pre-training on retinal waves significantly improves performance on tasks that test object invariance to spatial translation, while slightly improving performance on more complex tasks like image classification. Notably, these performance boosts are realized on held-out natural images even though the pre-training procedure does not include any natural image data. We then propose a geometrical explanation for the increase in network performance, namely that the spatiotemporal characteristics of retinal waves facilitate the formation of separable feature representations. In particular, we demonstrate that networks pre-trained on retinal waves are more effective at separating image manifolds than randomly initialized networks, especially for manifolds defined by sets of spatial translations. These findings indicate that the broad spatiotemporal properties of retinal waves prepare networks for higher order feature extraction

Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses

Despite the fact that the loss functions of deep neural networks are highly non-convex, gradient-based optimization algorithms converge to approximately the same performance from many random initial points. One thread of work has focused on explaining this phenomenon by characterizing the local curvature near critical points of the loss function, where the gradients are near zero, and demonstrating that neural network losses enjoy a no-bad-local-minima property and an abundance of saddle points. We report here that the methods used to find these putative critical points suffer from a bad local minima problem of their own: they often converge to or pass through regions where the gradient norm has a stationary point. We call these gradient-flat regions, since they arise when the gradient is approximately in the kernel of the Hessian, such that the loss is locally approximately linear, or flat, in the direction of the gradient. We describe how the presence of these regions necessitates care in both interpreting past results that claimed to find critical points of neural network losses and in designing second-order methods for optimizing neural networks.Comment: 18 pages, 5 figure

Clinical relevance of proteomic profiling in de novo pediatric acute myeloid leukemia:a Children’s Oncology Group study

Pediatric acute myeloid leukemia (AML) remains a fatal disease for at least 30% of patients, stressing the need for improved therapies and better risk stratification. As proteins are the unifying feature of (epi)genetic and environmental alterations, and are often targeted by novel chemotherapeutic agents, we studied the proteomic landscape of pediatric AML. Protein expression and activation levels were measured in 500 bulk leukemic patients’ samples and 30 control CD34(+) cell samples, using reverse phase protein arrays with 296 strictly validated antibodies. The multistep MetaGalaxy analysis methodology was applied and identified nine protein expression signatures (PrSIG), based on strong recurrent protein expression patterns. PrSIG were associated with cytogenetics and mutational state, and with favorable or unfavorable prognosis. Analysis based on treatment (i.e., ADE vs. ADE plus bortezomib) identified three PrSIG that did better with ADE plus bortezomib than with ADE alone. When PrSIG were studied in the context of cytogenetic risk groups, PrSIG were independently prognostic after multivariate analysis, suggesting a potential value for proteomics in combination with current classification systems. Proteins with universally increased (n=7) or decreased (n=17) expression were observed across PrSIG. Certain proteins significantly differentially expressed from normal could be identified, forming a hypothetical platform for personalized medicine

Modeling Visual Cortical Development

Representation is a critical component of visual neuroscience. While there is an extensive body of literature on the nature of visual representations, we lack a set of guiding principles for understanding how representations are learned during development. Our analysis here focuses on this question at a computational level. The first set of results addresses how representations are learned under the assumption of a sparse prior on the data. It is well known that sparse coding models trained on natural images learn basis functions whose shapes resemble the receptive fields (RFs) of simple cells in the primary visual cortex (V1). However, it is unclear whether certain types of basis functions emerge more quickly than others, or whether they develop simultaneously. We train an overcomplete sparse coding model (Sparsenet) on natural images and find that there is a spectral bias in the order of development of its basis functions, with basis functions tuned to lower spatial frequencies emerging earlier and higher spatial frequency basis functions emerging later. We observe the same trend in a biologically plausible sparse coding model (SAILnet) that uses leaky integrate-and-fire neurons and synaptically local learning rules, suggesting that this result is a general feature of sparse coding. These results are consistent with recent experimental evidence that the distribution of optimal stimuli for driving neurons to fire shifts towards higher frequencies during normal development in mouse V1. We find that the input data statistics can fully account for the spectral bias in sparse coding, and propose that visual experience is sufficient to drive the spectral bias in receptive field development. Our analysis of sparse coding models during training yields experimentally testable predictions for V1 development. In the next set of results, we investigate the potential for innately generated neural activity to drive the development of efficient representation in the visual cortex. Prior to the onset of vision, neurons in the developing mammalian retina spontaneously fire in correlated activity patterns known as retinal waves. Experimental evidence suggests that retinal waves strongly influence the emergence of sensory representations before visual experience. We model this early stage of functional development by using movies of neurally active developing retinas as pre-training data for neural networks. Specifically, we use unsupervised learning to train models on movies of retinal waves, then evaluate its performance on image classification tasks. We find that pre-training on retinal waves significantly improves performance on tasks that test object invariance to spatial translation, while slightly improving performance on more complex tasks like image classification. Notably, these performance boosts are realized on held-out natural images even though the pre-training procedure does not include any natural image data. We then propose a geometrical explanation for the increase in network performance, namely that the spatiotemporal characteristics of retinal waves facilitate the formation of separable feature representations. In particular, we demonstrate that networks pre-trained on retinal waves are more effective at separating image manifolds than randomly initialized networks, especially for manifolds defined by sets of spatial translations. These findings indicate that the broad spatiotemporal properties of retinal waves prepare networks for higher order feature extraction

Critical Point-Finding Methods Reveal Gradient-Flat Regions of Deep Network Losses

Despite the fact that the loss functions of deep neural networks are highly nonconvex, gradient-based optimization algorithms converge to approximately the same performance from many random initial points. One thread of work has focused on explaining this phenomenon by numerically characterizing the local curvature near critical points of the loss function, where the gradients are near zero. Such studies have reported that neural network losses enjoy a no-bad-local-minima property, in disagreement with more recent theoretical results. We report here that the methods used to find these putative critical points suffer from a bad local minima problem of their own: they often converge to or pass through regions where the gradient norm has a stationary point. We call these gradient-flat regions, since they arise when the gradient is approximately in the kernel of the Hessian, such that the loss is locally approximately linear, or flat, in the direction of the gradient. We describe how the presence of these regions necessitates care in both interpreting past results that claimed to find critical points of neural network losses and in designing second-order methods for optimizing neural networks

Recurrent Patterns of Protein Expression Signatures in Pediatric Acute Lymphoblastic Leukemia:Recognition and Therapeutic Guidance

Pediatric acute lymphoblastic leukemia (ALL) is the most common pediatric malignancy, and the second leading cause of pediatric cancer-related death in developed countries. While the cure rate for newly diagnosed ALL is excellent, the genetic heterogeneity and chemoresistance of leukemia cells at relapse makes individualized curative treatment plans difficult. We hypothesize that genetic events would coalesce into a finite number of protein signatures that could guide the design of individualized therapy. Custom reverse-phase protein arrays were produced from pediatric ALL (n = 73) and normal CD34(+) (n = 10) samples with 194 validated antibodies. Proteins were allocated into 31 protein functional groups (PFG) to analyze them in the context of other proteins, based on known associations from the literature. The optimal number of protein clusters was determined for each PFG. Protein networks showed distinct transition states, revealing "normal-like" and "leukemia-specific" protein patterns. Block clustering identified strong correlation between various protein clusters that formed 10 protein constellations. Patients that expressed similar recurrent combinations of constellations comprised 7 distinct signatures, correlating with risk stratification, cytogenetics, and laboratory features. Most constellations and signatures were specific for T-cell ALL or pre-B-cell ALL; however, some constellations showed significant overlap. Several signatures were associated with Hispanic ethnicity, suggesting that ethnic pathophysiologic differences likely exist. In addition, some constellations were enriched for "normal-like" protein clusters, whereas others had exclusively "leukemia-specific" patterns. (C) 2018 AACR

Recognition of Recurrent Protein Expression Patterns in Pediatric Acute Myeloid Leukemia Identified New Therapeutic Targets

Heterogeneity in the genetic landscape of pediatric acute myeloid leukemia (AML) makes personalized medicine challenging. As genetic events are mediated by the expression and function of proteins, recognition of recurrent protein patterns could enable classification of pediatric AML patients and could reveal crucial protein dependencies. This could help to rationally select combinations of therapeutic targets. To determine whether protein expression levels could be clustered into functionally relevant groups, custom reverse-phase protein arrays were performed on pediatric AML (n = 95) and CD34(+) normal bone marrow (n = 10) clinical specimens using 194 validated antibodies. To analyze proteins in the context of other proteins, all proteins were assembled into 31 protein functional groups (PFG). For each PFG, an optimal number of protein clusters was defined that represented distinct transition states. Block clustering analysis revealed strong correlations between various protein clusters and identified the existence of 12 protein constellations stratifying patients into 8 protein signatures. Signatures were correlated with therapeutic outcome, as well as certain laboratory and demographic characteristics. Comparison of acute lymphoblastic leukemia specimens from the same array and AML pediatric patient specimens demonstrated disease-specific signatures, but also identified the existence of shared constellations, suggesting joint protein deregulation between the diseases. (C) 2018 AACR

Clinical relevance of proteomic profiling in de novo pediatric acute myeloid leukemia: a Children’s Oncology Group study

Pediatric acute myeloid leukemia (AML) remains a fatal disease for at least 30% of patients, stressing the need for improved therapies and better risk stratification. As proteins are the unifying feature of (epi)genetic and environmental alterations, and are often targeted by novel chemotherapeutic agents, we studied the proteomic landscape of pediatric AML. Protein expression and activation levels were measured in 500 bulk leukemic patients’ samples and 30 control CD34+ cell samples, using reverse phase protein arrays with 296 strictly validated antibodies. The multistep MetaGalaxy analysis methodology was applied and identified nine protein expression signatures (PrSIG), based on strong recurrent protein expression patterns. PrSIG were associated with cytogenetics and mutational state, and with favorable or unfavorable prognosis. Analysis based on treatment (i.e., ADE vs. ADE plus bortezomib) identified three PrSIG that did better with ADE plus bortezomib than with ADE alone. When PrSIG were studied in the context of cytogenetic risk groups, PrSIG were independently prognostic after multivariate analysis, suggesting a potential value for proteomics in combination with current classification systems. Proteins with universally increased (n=7) or decreased (n=17) expression were observed across PrSIG. Certain proteins significantly differentially expressed from normal could be identified, forming a hypothetical platform for personalized medicine