95 research outputs found
Group Invariant Global Pooling
Much work has been devoted to devising architectures that build
group-equivariant representations, while invariance is often induced using
simple global pooling mechanisms. Little work has been done on creating
expressive layers that are invariant to given symmetries, despite the success
of permutation invariant pooling in various molecular tasks. In this work, we
present Group Invariant Global Pooling (GIGP), an invariant pooling layer that
is provably sufficiently expressive to represent a large class of invariant
functions. We validate GIGP on rotated MNIST and QM9, showing improvements for
the latter while attaining identical results for the former. By making the
pooling process group orbit-aware, this invariant aggregation method leads to
improved performance, while performing well-principled group aggregation
Benchmarking Graph Neural Networks
Graph neural networks (GNNs) have become the standard toolkit for analyzing
and learning from data on graphs. As the field grows, it becomes critical to
identify key architectures and validate new ideas that generalize to larger,
more complex datasets. Unfortunately, it has been increasingly difficult to
gauge the effectiveness of new models in the absence of a standardized
benchmark with consistent experimental settings. In this paper, we introduce a
reproducible GNN benchmarking framework, with the facility for researchers to
add new models conveniently for arbitrary datasets. We demonstrate the
usefulness of our framework by presenting a principled investigation into the
recent Weisfeiler-Lehman GNNs (WL-GNNs) compared to message passing-based graph
convolutional networks (GCNs) for a variety of graph tasks, i.e. graph
regression/classification and node/link prediction, with medium-scale datasets.Comment: Benchmarking framework on GitHub at
https://github.com/graphdeeplearning/benchmarking-gnn
On the Expressive Power of Geometric Graph Neural Networks
The expressive power of Graph Neural Networks (GNNs) has been studied
extensively through the Weisfeiler-Leman (WL) graph isomorphism test. However,
standard GNNs and the WL framework are inapplicable for geometric graphs
embedded in Euclidean space, such as biomolecules, materials, and other
physical systems. In this work, we propose a geometric version of the WL test
(GWL) for discriminating geometric graphs while respecting the underlying
physical symmetries: permutations, rotation, reflection, and translation. We
use GWL to characterise the expressive power of geometric GNNs that are
invariant or equivariant to physical symmetries in terms of distinguishing
geometric graphs. GWL unpacks how key design choices influence geometric GNN
expressivity: (1) Invariant layers have limited expressivity as they cannot
distinguish one-hop identical geometric graphs; (2) Equivariant layers
distinguish a larger class of graphs by propagating geometric information
beyond local neighbourhoods; (3) Higher order tensors and scalarisation enable
maximally powerful geometric GNNs; and (4) GWL's discrimination-based
perspective is equivalent to universal approximation. Synthetic experiments
supplementing our results are available at
https://github.com/chaitjo/geometric-gnn-dojoComment: NeurIPS 2022 Workshop on Symmetry and Geometry in Neural
Representation
Point Discriminative Learning for Data-efficient 3D Point Cloud Analysis
3D point cloud analysis has drawn a lot of research attention due to its wide
applications. However, collecting massive labelled 3D point cloud data is both
time-consuming and labor-intensive. This calls for data-efficient learning
methods. In this work we propose PointDisc, a point discriminative learning
method to leverage self-supervisions for data-efficient 3D point cloud
classification and segmentation. PointDisc imposes a novel point discrimination
loss on the middle and global level features produced by the backbone network.
This point discrimination loss enforces learned features to be consistent with
points belonging to the corresponding local shape region and inconsistent with
randomly sampled noisy points. We conduct extensive experiments on 3D object
classification, 3D semantic and part segmentation, showing the benefits of
PointDisc for data-efficient learning. Detailed analysis demonstrate that
PointDisc learns unsupervised features that well capture local and global
geometry.Comment: This work is published in 3DV 202
On Representation Knowledge Distillation for Graph Neural Networks
Knowledge distillation is a learning paradigm for boosting resource-efficient
graph neural networks (GNNs) using more expressive yet cumbersome teacher
models. Past work on distillation for GNNs proposed the Local Structure
Preserving loss (LSP), which matches local structural relationships defined
over edges across the student and teacher's node embeddings. This paper studies
whether preserving the global topology of how the teacher embeds graph data can
be a more effective distillation objective for GNNs, as real-world graphs often
contain latent interactions and noisy edges. We propose Graph Contrastive
Representation Distillation (G-CRD), which uses contrastive learning to
implicitly preserve global topology by aligning the student node embeddings to
those of the teacher in a shared representation space. Additionally, we
introduce an expanded set of benchmarks on large-scale real-world datasets
where the performance gap between teacher and student GNNs is non-negligible.
Experiments across 4 datasets and 14 heterogeneous GNN architectures show that
G-CRD consistently boosts the performance and robustness of lightweight GNNs,
outperforming LSP (and a global structure preserving variant of LSP) as well as
baselines from 2D computer vision. An analysis of the representational
similarity among teacher and student embedding spaces reveals that G-CRD
balances preserving local and global relationships, while structure preserving
approaches are best at preserving one or the other
- …