Input-driven unsupervised learning in recurrent neural networks

Understanding the theoretical foundations of how memories are encoded and retrieved in neural populations is a central challenge in neuroscience. A popular theoretical scenario for modeling memory function is an attractor neural network with Hebbian learning (e.g. the Hopfield model). The model simplicity and the locality of the synaptic update rules come at the cost of a limited storage capacity, compared with the capacity achieved with supervised learning algorithms, whose biological plausibility is questionable. Here, we present an on-line learning rule for a recurrent neural network that achieves near-optimal performance without an explicit supervisory error signal and using only locally accessible information, and which is therefore biologically plausible. The fully connected network consists of excitatory units with plastic recurrent connections and non-plastic inhibitory feedback stabilizing the network dynamics; the patterns to be memorized are presented on-line as strong afferent currents, producing a bimodal distribution for the neuron synaptic inputs ('local fields'). Synapses corresponding to active inputs are modified as a function of the position of the local field with respect to three thresholds. Above the highest threshold, and below the lowest threshold, no plasticity occurs. In between these two thresholds, potentiation/depression occurs when the local field is above/below an intermediate threshold. An additional parameter of the model allows to trade storage capacity for robustness, i.e. increased size of the basins of attraction. We simulated a network of 1001 excitatory neurons implementing this rule and measured its storage capacity for different sizes of the basins of attraction: our results show that, for any given basin size, our network more than doubles the storage capacity, compared with a standard Hopfield network. Our learning rule is consistent with available experimental data documenting how plasticity depends on firing rate. It predicts that at high enough firing rates, no potentiation should occu

Input-driven unsupervised learning in recurrent neural networks

Understanding the theoretical foundations of how memories are encoded and retrieved in neural populations is a central challenge in neuroscience. A popular theoretical scenario for modeling memory function is an attractor neural network with Hebbian learning (e.g. the Hopfield model). The model simplicity and the locality of the synaptic update rules come at the cost of a limited storage capacity, compared with the capacity achieved with supervised learning algorithms, whose biological plausibility is questionable. Here, we present an on-line learning rule for a recurrent neural network that achieves near-optimal performance without an explicit supervisory error signal and using only locally accessible information, and which is therefore biologically plausible. The fully connected network consists of excitatory units with plastic recurrent connections and non-plastic inhibitory feedback stabilizing the network dynamics; the patterns to be memorized are presented on-line as strong afferent currents, producing a bimodal distribution for the neuron synaptic inputs (’local fields’). Synapses corresponding to active inputs are modified as a function of the position of the local field with respect to three thresholds. Above the highest threshold, and below the lowest threshold, no plasticity occurs. In between these two thresholds, potentiation/depression occurs when the local field is above/below an intermediate threshold. An additional parameter of the model allows to trade storage capacity for robustness, i.e. increased size of the basins of attraction. We simulated a network of 1001 excitatory neurons implementing this rule and measured its storage capacity for different sizes of the basins of attraction: our results show that, for any given basin size, our network more than doubles the storage capacity, compared with a standard Hopfield network. Our learning rule is consistent with available experimental data documenting how plasticity depends on firing rate. It predicts that at high enough firing rates, no potentiation should occur

Enhancing VAEs for Collaborative Filtering: Flexible Priors & Gating Mechanisms

Neural network based models for collaborative filtering have started to gain attention recently. One branch of research is based on using deep generative models to model user preferences where variational autoencoders were shown to produce state-of-the-art results. However, there are some potentially problematic characteristics of the current variational autoencoder for CF. The first is the too simplistic prior that VAEs incorporate for learning the latent representations of user preference. The other is the model's inability to learn deeper representations with more than one hidden layer for each network. Our goal is to incorporate appropriate techniques to mitigate the aforementioned problems of variational autoencoder CF and further improve the recommendation performance. Our work is the first to apply flexible priors to collaborative filtering and show that simple priors (in original VAEs) may be too restrictive to fully model user preferences and setting a more flexible prior gives significant gains. We experiment with the VampPrior, originally proposed for image generation, to examine the effect of flexible priors in CF. We also show that VampPriors coupled with gating mechanisms outperform SOTA results including the Variational Autoencoder for Collaborative Filtering by meaningful margins on 2 popular benchmark datasets (MovieLens & Netflix)

Weakly-supervised learning for image-based classification of primary melanomas into genomic immune subgroups

Determining early-stage prognostic markers and stratifying patients for effective treatment are two key challenges for improving outcomes for melanoma patients. Previous studies have used tumour transcriptome data to stratify patients into immune subgroups, which were associated with differential melanoma specific survival and potential treatment strategies. However, acquiring transcriptome data is a time-consuming and costly process. Moreover, it is not routinely used in the current clinical workflow. Here we attempt to overcome this by developing deep learning models to classify gigapixel H&E stained pathology slides, which are well established in clinical workflows, into these immune subgroups. Previous subtyping approaches have employed supervised learning which requires fully annotated data, or have only examined single genetic mutations in melanoma patients. We leverage a multiple-instance learning approach, which only requires slide-level labels and uses an attention mechanism to highlight regions of high importance to the classification. Moreover, we show that pathology-specific self-supervised models generate better representations compared to pathology-agnostic models for improving our model performance, achieving a mean AUC of 0.76 for classifying histopathology images as high or low immune subgroups. We anticipate that this method may allow us to find new biomarkers of high importance and could act as a tool for clinicians to infer the immune landscape of tumours and stratify patients, without needing to carry out additional expensive genetic tests

Shape similarity, better than semantic membership, accounts for the structure of visual object representations in a population of monkey inferotemporal neurons

The anterior inferotemporal cortex (IT) is the highest stage along the hierarchy of visual areas that, in primates, processes visual objects. Although several lines of evidence suggest that IT primarily represents visual shape information, some recent studies have argued that neuronal ensembles in IT code the semantic membership of visual objects (i.e., represent conceptual classes such as animate and inanimate objects). In this study, we investigated to what extent semantic, rather than purely visual information, is represented in IT by performing a multivariate analysis of IT responses to a set of visual objects. By relying on a variety of machine-learning approaches (including a cutting-edge clustering algorithm that has been recently developed in the domain of statistical physics), we found that, in most instances, IT representation of visual objects is accounted for by their similarity at the level of shape or, more surprisingly, low-level visual properties. Only in a few cases we observed IT representations of semantic classes that were not explainable by the visual similarity of their members. Overall, these findings reassert the primary function of IT as a conveyor of explicit visual shape information, and reveal that low-level visual properties are represented in IT to a greater extent than previously appreciated. In addition, our work demonstrates how combining a variety of state-of-the-art multivariate approaches, and carefully estimating the contribution of shape similarity to the representation of object categories, can substantially advance our understanding of neuronal coding of visual objects in cortex

Collaborative learning about e-health for mental health professionals and service users in a structured anonymous online short course: pilot study.

BACKGROUND: Professionals are interested in using e-health but implementation of new methods is slow. Barriers to implementation include the need for training and limited awareness or experience. Research may not always convince mental health professionals (MHPs). Adding the 'voice' of mental health service users (MHSUs) in collaborative learning may help. Involving MHSUs in face-face education can be difficult. We had previously been unable to engage MHPs in online discussion with MHSUs. Here we assessed the feasibility of short online courses involving MHSUs and MHPs. METHODS: We ran three e-health courses, comprising live interactive webcast, week's access to a discussion forum, and final live interactive webcast. We recruited MHPs via posters, newsletters, and telephone from a local NHS trust, and online via mailing lists and personal contacts from NHS trusts and higher education. We recruited MHSUs via a previous project and an independent user involvement service. Participants were presented with research evidence about e-health and asked to discuss topics using professional and lived experience. Feasibility was assessed through recruitment and attrition, participation, and researcher workloads. Outcomes of self-esteem and general self-efficacy (MHSUs), and Internet self-efficacy and confidence (MHPs) were piloted. RESULTS: Online recruiting was effective. We lost 15/41 from registration to follow-up but only 5/31 that participated in the course failed to complete follow-up. Nineteen MHPs and 12 MHSUs took part and engaged with each other in online discussion. Feedback was positive; three-quarters of MHPs indicated future plans to use the Internet for practice, and 80% of MHSUs felt the course should be continued. Running three courses for 31 participants took between 200 to 250 hours. Before and after outcome measures were completed by 26/31 that participated. MHP Internet self-efficacy and general Internet confidence, MHSU self-esteem and general self-efficacy, all seemed reliable and seemed to show some increase. CONCLUSIONS: Collaborative learning between MHSUs and MHPs in a structured online anonymous environment over a one-week course is feasible, may be more practical and less costly than face-face methods, and is worthy of further study

Gene Expression in a Drosophila Model of Mitochondrial Disease

Background A point mutation in the Drosophila gene technical knockout (tko), encoding mitoribosomal protein S12, was previously shown to cause a phenotype of respiratory chain deficiency, developmental delay, and neurological abnormalities similar to those presented in many human mitochondrial disorders, as well as defective courtship behavior. Methodology/Principal Findings Here, we describe a transcriptome-wide analysis of gene expression in tko25t mutant flies that revealed systematic and compensatory changes in the expression of genes connected with metabolism, including up-regulation of lactate dehydrogenase and of many genes involved in the catabolism of fats and proteins, and various anaplerotic pathways. Gut-specific enzymes involved in the primary mobilization of dietary fats and proteins, as well as a number of transport functions, were also strongly up-regulated, consistent with the idea that oxidative phosphorylation OXPHOS dysfunction is perceived physiologically as a starvation for particular biomolecules. In addition, many stress-response genes were induced. Other changes may reflect a signature of developmental delay, notably a down-regulation of genes connected with reproduction, including gametogenesis, as well as courtship behavior in males; logically this represents a programmed response to a mitochondrially generated starvation signal. The underlying signalling pathway, if conserved, could influence many physiological processes in response to nutritional stress, although any such pathway involved remains unidentified. Conclusions/Significance These studies indicate that general and organ-specific metabolism is transformed in response to mitochondrial dysfunction, including digestive and absorptive functions, and give important clues as to how novel therapeutic strategies for mitochondrial disorders might be developed.Public Library of Scienc

Global, regional, and national burden of diabetes from 1990 to 2021, with projections of prevalence to 2050: a systematic analysis for the Global Burden of Disease Study 2021

Background: Diabetes is one of the leading causes of death and disability worldwide, and affects people regardless of country, age group, or sex. Using the most recent evidentiary and analytical framework from the Global Burden of Diseases, Injuries, and Risk Factors Study (GBD), we produced location-specific, age-specific, and sex-specific estimates of diabetes prevalence and burden from 1990 to 2021, the proportion of type 1 and type 2 diabetes in 2021, the proportion of the type 2 diabetes burden attributable to selected risk factors, and projections of diabetes prevalence through 2050. Methods: Estimates of diabetes prevalence and burden were computed in 204 countries and territories, across 25 age groups, for males and females separately and combined; these estimates comprised lost years of healthy life, measured in disability-adjusted life-years (DALYs; defined as the sum of years of life lost [YLLs] and years lived with disability [YLDs]). We used the Cause of Death Ensemble model (CODEm) approach to estimate deaths due to diabetes, incorporating 25 666 location-years of data from vital registration and verbal autopsy reports in separate total (including both type 1 and type 2 diabetes) and type-specific models. Other forms of diabetes, including gestational and monogenic diabetes, were not explicitly modelled. Total and type 1 diabetes prevalence was estimated by use of a Bayesian meta-regression modelling tool, DisMod-MR 2.1, to analyse 1527 location-years of data from the scientific literature, survey microdata, and insurance claims; type 2 diabetes estimates were computed by subtracting type 1 diabetes from total estimates. Mortality and prevalence estimates, along with standard life expectancy and disability weights, were used to calculate YLLs, YLDs, and DALYs. When appropriate, we extrapolated estimates to a hypothetical population with a standardised age structure to allow comparison in populations with different age structures. We used the comparative risk assessment framework to estimate the risk-attributable type 2 diabetes burden for 16 risk factors falling under risk categories including environmental and occupational factors, tobacco use, high alcohol use, high body-mass index (BMI), dietary factors, and low physical activity. Using a regression framework, we forecast type 1 and type 2 diabetes prevalence through 2050 with Socio-demographic Index (SDI) and high BMI as predictors, respectively. Findings: In 2021, there were 529 million (95% uncertainty interval [UI] 500–564) people living with diabetes worldwide, and the global age-standardised total diabetes prevalence was 6·1% (5·8–6·5). At the super-region level, the highest age-standardised rates were observed in north Africa and the Middle East (9·3% [8·7–9·9]) and, at the regional level, in Oceania (12·3% [11·5–13·0]). Nationally, Qatar had the world's highest age-specific prevalence of diabetes, at 76·1% (73·1–79·5) in individuals aged 75–79 years. Total diabetes prevalence—especially among older adults—primarily reflects type 2 diabetes, which in 2021 accounted for 96·0% (95·1–96·8) of diabetes cases and 95·4% (94·9–95·9) of diabetes DALYs worldwide. In 2021, 52·2% (25·5–71·8) of global type 2 diabetes DALYs were attributable to high BMI. The contribution of high BMI to type 2 diabetes DALYs rose by 24·3% (18·5–30·4) worldwide between 1990 and 2021. By 2050, more than 1·31 billion (1·22–1·39) people are projected to have diabetes, with expected age-standardised total diabetes prevalence rates greater than 10% in two super-regions: 16·8% (16·1–17·6) in north Africa and the Middle East and 11·3% (10·8–11·9) in Latin America and Caribbean. By 2050, 89 (43·6%) of 204 countries and territories will have an age-standardised rate greater than 10%. Interpretation: Diabetes remains a substantial public health issue. Type 2 diabetes, which makes up the bulk of diabetes cases, is largely preventable and, in some cases, potentially reversible if identified and managed early in the disease course. However, all evidence indicates that diabetes prevalence is increasing worldwide, primarily due to a rise in obesity caused by multiple factors. Preventing and controlling type 2 diabetes remains an ongoing challenge. It is essential to better understand disparities in risk factor profiles and diabetes burden across populations, to inform strategies to successfully control diabetes risk factors within the context of multiple and complex drivers. Funding: Bill & Melinda Gates Foundation