85 research outputs found
Towards Faster Training of Global Covariance Pooling Networks by Iterative Matrix Square Root Normalization
Global covariance pooling in convolutional neural networks has achieved
impressive improvement over the classical first-order pooling. Recent works
have shown matrix square root normalization plays a central role in achieving
state-of-the-art performance. However, existing methods depend heavily on
eigendecomposition (EIG) or singular value decomposition (SVD), suffering from
inefficient training due to limited support of EIG and SVD on GPU. Towards
addressing this problem, we propose an iterative matrix square root
normalization method for fast end-to-end training of global covariance pooling
networks. At the core of our method is a meta-layer designed with loop-embedded
directed graph structure. The meta-layer consists of three consecutive
nonlinear structured layers, which perform pre-normalization, coupled matrix
iteration and post-compensation, respectively. Our method is much faster than
EIG or SVD based ones, since it involves only matrix multiplications, suitable
for parallel implementation on GPU. Moreover, the proposed network with ResNet
architecture can converge in much less epochs, further accelerating network
training. On large-scale ImageNet, we achieve competitive performance superior
to existing counterparts. By finetuning our models pre-trained on ImageNet, we
establish state-of-the-art results on three challenging fine-grained
benchmarks. The source code and network models will be available at
http://www.peihuali.org/iSQRT-COVComment: Accepted to CVPR 201
Grouping-matrix based Graph Pooling with Adaptive Number of Clusters
Graph pooling is a crucial operation for encoding hierarchical structures
within graphs. Most existing graph pooling approaches formulate the problem as
a node clustering task which effectively captures the graph topology.
Conventional methods ask users to specify an appropriate number of clusters as
a hyperparameter, then assume that all input graphs share the same number of
clusters. In inductive settings where the number of clusters can vary, however,
the model should be able to represent this variation in its pooling layers in
order to learn suitable clusters. Thus we propose GMPool, a novel
differentiable graph pooling architecture that automatically determines the
appropriate number of clusters based on the input data. The main intuition
involves a grouping matrix defined as a quadratic form of the pooling operator,
which induces use of binary classification probabilities of pairwise
combinations of nodes. GMPool obtains the pooling operator by first computing
the grouping matrix, then decomposing it. Extensive evaluations on molecular
property prediction tasks demonstrate that our method outperforms conventional
methods.Comment: 10 pages, 3 figure
Learning Linear Dynamical Systems via Spectral Filtering
We present an efficient and practical algorithm for the online prediction of
discrete-time linear dynamical systems with a symmetric transition matrix. We
circumvent the non-convex optimization problem using improper learning:
carefully overparameterize the class of LDSs by a polylogarithmic factor, in
exchange for convexity of the loss functions. From this arises a
polynomial-time algorithm with a near-optimal regret guarantee, with an
analogous sample complexity bound for agnostic learning. Our algorithm is based
on a novel filtering technique, which may be of independent interest: we
convolve the time series with the eigenvectors of a certain Hankel matrix.Comment: Published as a conference paper at NIPS 201
New Deep Neural Networks for Unsupervised Feature Learning on Graph Data
Graph data are ubiquitous in the real world, such as social networks, biological networks. To analyze graph data, a fundamental task is to learn node features to benefit downstream tasks, such as node classification, community detection. Inspired by the powerful feature learning capability of deep neural networks on various tasks, it is important and necessary to explore deep neural networks for feature learning on graphs. Different from the regular image and sequence data, graph data encode the complicated relational information between different nodes, which challenges the classical deep neural networks. Moreover, in real-world applications, the label of nodes in graph data is usually not available, which makes the feature learning on graphs more difficult.
To address these challenging issues, this thesis is focusing on designing new deep neural networks to effectively explore the relational information for unsupervised feature learning on graph data.
First, to address the sparseness issue of the relational information, I propose a new proximity generative adversarial network which can discover the underlying relational information for learning better node representations. Meanwhile, a new self-paced network embedding method is designed to address the unbalance issue of the relational information when learning node representations. Additionally, to deal with rich attributes associated to nodes, I develop a new deep neural network to capture various relational information in both topological structure and node attributes for enhancing network embedding. Furthermore, to preserve the relational information in the hidden layers of deep neural networks, I develop a novel graph convolutional neural network (GCN) based on conditional random fields, which is the first algorithm applying this kind of graphical models to graph neural networks in an unsupervised manner
3D Generative Model Latent Disentanglement via Local Eigenprojection
Designing realistic digital humans is extremely complex. Most data-driven generative models used to simplify the creation of their underlying geometric shape do not offer control over the generation of local shape attributes. In this paper, we overcome this limitation by introducing a novel loss function grounded in spectral geometry and applicable to different neural-network-based generative models of 3D head and body meshes. Encouraging the latent variables of mesh variational autoencoders (VAEs) or generative adversarial networks (GANs) to follow the local eigenprojections of identity attributes, we improve latent disentanglement and properly decouple the attribute creation. Experimental results show that our local eigenprojection disentangled (LED) models not only offer improved disentanglement with respect to the state-of-the-art, but also maintain good generation capabilities with training times comparable to the vanilla implementations of the models. Our code and pre-trained models are available at github.com/simofoti/LocalEigenprojDisentangled
- …