66,367 research outputs found
Convolutional Kernel Networks
An important goal in visual recognition is to devise image representations
that are invariant to particular transformations. In this paper, we address
this goal with a new type of convolutional neural network (CNN) whose
invariance is encoded by a reproducing kernel. Unlike traditional approaches
where neural networks are learned either to represent data or for solving a
classification task, our network learns to approximate the kernel feature map
on training data. Such an approach enjoys several benefits over classical ones.
First, by teaching CNNs to be invariant, we obtain simple network architectures
that achieve a similar accuracy to more complex ones, while being easy to train
and robust to overfitting. Second, we bridge a gap between the neural network
literature and kernels, which are natural tools to model invariance. We
evaluate our methodology on visual recognition tasks where CNNs have proven to
perform well, e.g., digit recognition with the MNIST dataset, and the more
challenging CIFAR-10 and STL-10 datasets, where our accuracy is competitive
with the state of the art.Comment: appears in Advances in Neural Information Processing Systems (NIPS),
Dec 2014, Montreal, Canada, http://nips.c
Irregular Convolutional Neural Networks
Convolutional kernels are basic and vital components of deep Convolutional
Neural Networks (CNN). In this paper, we equip convolutional kernels with shape
attributes to generate the deep Irregular Convolutional Neural Networks (ICNN).
Compared to traditional CNN applying regular convolutional kernels like
, our approach trains irregular kernel shapes to better fit the
geometric variations of input features. In other words, shapes are learnable
parameters in addition to weights. The kernel shapes and weights are learned
simultaneously during end-to-end training with the standard back-propagation
algorithm. Experiments for semantic segmentation are implemented to validate
the effectiveness of our proposed ICNN.Comment: 7 pages, 5 figures, 3 table
Group Invariance, Stability to Deformations, and Complexity of Deep Convolutional Representations
The success of deep convolutional architectures is often attributed in part
to their ability to learn multiscale and invariant representations of natural
signals. However, a precise study of these properties and how they affect
learning guarantees is still missing. In this paper, we consider deep
convolutional representations of signals; we study their invariance to
translations and to more general groups of transformations, their stability to
the action of diffeomorphisms, and their ability to preserve signal
information. This analysis is carried by introducing a multilayer kernel based
on convolutional kernel networks and by studying the geometry induced by the
kernel mapping. We then characterize the corresponding reproducing kernel
Hilbert space (RKHS), showing that it contains a large class of convolutional
neural networks with homogeneous activation functions. This analysis allows us
to separate data representation from learning, and to provide a canonical
measure of model complexity, the RKHS norm, which controls both stability and
generalization of any learned model. In addition to models in the constructed
RKHS, our stability analysis also applies to convolutional networks with
generic activations such as rectified linear units, and we discuss its
relationship with recent generalization bounds based on spectral norms
Kernel Graph Convolutional Neural Networks
Graph kernels have been successfully applied to many graph classification
problems. Typically, a kernel is first designed, and then an SVM classifier is
trained based on the features defined implicitly by this kernel. This two-stage
approach decouples data representation from learning, which is suboptimal. On
the other hand, Convolutional Neural Networks (CNNs) have the capability to
learn their own features directly from the raw data during training.
Unfortunately, they cannot handle irregular data such as graphs. We address
this challenge by using graph kernels to embed meaningful local neighborhoods
of the graphs in a continuous vector space. A set of filters is then convolved
with these patches, pooled, and the output is then passed to a feedforward
network. With limited parameter tuning, our approach outperforms strong
baselines on 7 out of 10 benchmark datasets.Comment: Accepted at ICANN '1
End-to-End Kernel Learning with Supervised Convolutional Kernel Networks
In this paper, we introduce a new image representation based on a multilayer
kernel machine. Unlike traditional kernel methods where data representation is
decoupled from the prediction task, we learn how to shape the kernel with
supervision. We proceed by first proposing improvements of the
recently-introduced convolutional kernel networks (CKNs) in the context of
unsupervised learning; then, we derive backpropagation rules to take advantage
of labeled training data. The resulting model is a new type of convolutional
neural network, where optimizing the filters at each layer is equivalent to
learning a linear subspace in a reproducing kernel Hilbert space (RKHS). We
show that our method achieves reasonably competitive performance for image
classification on some standard "deep learning" datasets such as CIFAR-10 and
SVHN, and also for image super-resolution, demonstrating the applicability of
our approach to a large variety of image-related tasks.Comment: to appear in Advances in Neural Information Processing Systems (NIPS
Kernel Normalized Convolutional Networks
Existing deep convolutional neural network (CNN) architectures frequently
rely upon batch normalization (BatchNorm) to effectively train the model.
BatchNorm significantly improves model performance in centralized training, but
it is unsuitable for federated learning and differential privacy settings. Even
in centralized learning, BatchNorm performs poorly with smaller batch sizes. To
address these limitations, we propose kernel normalization and kernel
normalized convolutional layers, and incorporate them into kernel normalized
convolutional networks (KNConvNets) as the main building blocks. We implement
KNConvNets corresponding to the state-of-the-art CNNs such as VGGNets and
ResNets while forgoing BatchNorm layers. Through extensive experiments, we
illustrate KNConvNets consistently outperform their batch, group, and layer
normalized counterparts in terms of both accuracy and convergence rate in
centralized, federated, and differentially private learning settings
Convolutional Kernel Networks for Graph-Structured Data
We introduce a family of multilayer graph kernels and establish new links
between graph convolutional neural networks and kernel methods. Our approach
generalizes convolutional kernel networks to graph-structured data, by
representing graphs as a sequence of kernel feature maps, where each node
carries information about local graph substructures. On the one hand, the
kernel point of view offers an unsupervised, expressive, and easy-to-regularize
data representation, which is useful when limited samples are available. On the
other hand, our model can also be trained end-to-end on large-scale data,
leading to new types of graph convolutional neural networks. We show that our
method achieves competitive performance on several graph classification
benchmarks, while offering simple model interpretation. Our code is freely
available at https://github.com/claying/GCKN
- …