124 research outputs found
What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoders
The last years have witnessed the emergence of a promising self-supervised
learning strategy, referred to as masked autoencoding. However, there is a lack
of theoretical understanding of how masking matters on graph autoencoders
(GAEs). In this work, we present masked graph autoencoder (MaskGAE), a
self-supervised learning framework for graph-structured data. Different from
standard GAEs, MaskGAE adopts masked graph modeling (MGM) as a principled
pretext task - masking a portion of edges and attempting to reconstruct the
missing part with partially visible, unmasked graph structure. To understand
whether MGM can help GAEs learn better representations, we provide both
theoretical and empirical evidence to comprehensively justify the benefits of
this pretext task. Theoretically, we establish close connections between GAEs
and contrastive learning, showing that MGM significantly improves the
self-supervised learning scheme of GAEs. Empirically, we conduct extensive
experiments on a variety of graph benchmarks, demonstrating the superiority of
MaskGAE over several state-of-the-arts on both link prediction and node
classification tasks.Comment: KDD 2023 research track. Code available at
https://github.com/EdisonLeeeee/MaskGA
Approximations of Shannon Mutual Information for Discrete Variables with Applications to Neural Population Coding
Although Shannon mutual information has been widely used, its effective
calculation is often difficult for many practical problems, including those in
neural population coding. Asymptotic formulas based on Fisher information
sometimes provide accurate approximations to the mutual information but this
approach is restricted to continuous variables because the calculation of
Fisher information requires derivatives with respect to the encoded variables.
In this paper, we consider information-theoretic bounds and approximations of
the mutual information based on Kullback--Leibler divergence and R\'{e}nyi
divergence. We propose several information metrics to approximate Shannon
mutual information in the context of neural population coding. While our
asymptotic formulas all work for discrete variables, one of them has consistent
performance and high accuracy regardless of whether the encoded variables are
discrete or continuous. We performed numerical simulations and confirmed that
our approximation formulas were highly accurate for approximating the mutual
information between the stimuli and the responses of a large neural population.
These approximation formulas may potentially bring convenience to the
applications of information theory to many practical and theoretical problems.Comment: 31 pages, 6 figure
Recommended from our members
Mixed Selectivity via Unsupervised Learning in Neural Networks
Mixed selectivity characterises neurons that simultaneously respond to different input stimuli. Neurons with mixed selectivity have been observed in multiple brain regions, and are hypothesised to play important roles in neural computation. Recently, both experimental and theoretical work demonstrated the importance of mixed selectivity in context-dependent decision tasks. This thesis extends existing theoretical work on mixed selectivity, arguing for a general and statistical role of mixed selectivity in learning complex dependencies of input stimuli. This role can be motivated from unsupervised learning of generative models, and is exhibited in increased mutual information between the stimuli and their neural representation. This argument is supported empirically using simulation of a sequence disambiguation task that incorporated key aspects of related behaviour experiments. Mixed selectivity neurons that resembled hippocampal place cells were emerged from models optimised only for behaviour. To understand these results, as well as to generalise the findings to a wider range of computations, I provided a formal connection between learning robust models and mixed selectivity
Backwards is the way forward: feedback in the cortical hierarchy predicts the expected future
Clark offers a powerful description of the brain as a prediction machine, which offers progress on two distinct levels. First, on an abstract conceptual level, it provides a unifying framework for perception, action, and cognition (including subdivisions such as attention, expectation, and imagination). Second, hierarchical prediction offers progress on a concrete descriptive level for testing and constraining conceptual elements and mechanisms of predictive coding models (estimation of predictions, prediction errors, and internal models)
Dimensionality reduction and unsupervised learning techniques applied to clinical psychiatric and neuroimaging phenotypes
Unsupervised learning and other multivariate analysis techniques are increasingly recognized in neuropsychiatric research. Here, finite mixture models and random forests were applied to clinical observations of patients with major depression to detect and validate treatment response subgroups. Further, independent component analysis and agglomerative hierarchical clustering were combined to build a brain parcellation solely on structural covariance information of magnetic resonance brain images. Übersetzte Kurzfassung: Unüberwachtes Lernen und andere multivariate Analyseverfahren werden zunehmend auf neuropsychiatrische Fragestellungen angewendet. Finite mixture Modelle wurden auf klinische Skalen von Patienten mit schwerer Depression appliziert, um Therapieantwortklassen zu bilden und mit Random Forests zu validieren. Unabhängigkeitsanalysen und agglomeratives hierarchisches Clustering wurden kombiniert, um die strukturelle Kovarianz von Magnetresonanztomographie-Bildern für eine Hirnparzellierung zu nutzen
- …