9,608 research outputs found
Generalization in Graph Neural Networks: Improved PAC-Bayesian Bounds on Graph Diffusion
Graph neural networks are widely used tools for graph prediction tasks.
Motivated by their empirical performance, prior works have developed
generalization bounds for graph neural networks, which scale with graph
structures in terms of the maximum degree. In this paper, we present
generalization bounds that instead scale with the largest singular value of the
graph neural network's feature diffusion matrix. These bounds are numerically
much smaller than prior bounds for real-world graphs. We also construct a lower
bound of the generalization gap that matches our upper bound asymptotically. To
achieve these results, we analyze a unified model that includes prior works'
settings (i.e., convolutional and message-passing networks) and new settings
(i.e., graph isomorphism networks). Our key idea is to measure the stability of
graph neural networks against noise perturbations using Hessians. Empirically,
we find that Hessian-based measurements correlate with the observed
generalization gaps of graph neural networks accurately. Optimizing noise
stability properties for fine-tuning pretrained graph neural networks also
improves test performance on several graph-level classification tasks.Comment: 36 pages, 2 tables, 3 figures. Appeared in AISTATS 202
On Lipschitz Regularization of Convolutional Layers using Toeplitz Matrix Theory
This paper tackles the problem of Lipschitz regularization of Convolutional
Neural Networks. Lipschitz regularity is now established as a key property of
modern deep learning with implications in training stability, generalization,
robustness against adversarial examples, etc. However, computing the exact
value of the Lipschitz constant of a neural network is known to be NP-hard.
Recent attempts from the literature introduce upper bounds to approximate this
constant that are either efficient but loose or accurate but computationally
expensive. In this work, by leveraging the theory of Toeplitz matrices, we
introduce a new upper bound for convolutional layers that is both tight and
easy to compute. Based on this result we devise an algorithm to train Lipschitz
regularized Convolutional Neural Networks
- …