577 research outputs found
Highway Network Block with Gates Constraints for Training Very Deep Networks
In this paper, we propose to reformulate the learning of the highway network block to realize both early optimization and improved generalization of very deep networks while preserving the network depth. Gate constraints are duly employed to improve optimization, latent representations and parameterization usage in order to efficiently learn hierarchical feature transformations which are crucial for the success of any deep network. One of the earliest
very deep models with over 30 layers that was successfully trained relied on highway network blocks. Although, highway blocks suffice for alleviating optimization problem via improved information flow, we show for the first time that further in training such highway blocks may result into learning mostly untransformed features and therefore a reduction in the effective depth of the model; this could negatively impact model generalization performance. Using the
proposed approach, 15-layer and 20-layer models are successfully trained with one gate and a 32-layer model using three gates. This leads to a drastic reduction of model parameters as compared to the original highway network. Extensive experiments on CIFAR-10, CIFAR-100, Fashion-MNIST and USPS datasets are performed to validate the effectiveness of the proposed approach. Particularly, we outperform the original highway network and many state-ofthe-
art results. To the best our knowledge, on the Fashion-MNIST and USPS datasets, the achieved results are the best reported in literature
Training Very Deep Networks via Residual Learning with Stochastic Input Shortcut Connections
Many works have posited the benefit of depth in deep networks. However,
one of the problems encountered in the training of very deep networks is feature
reuse; that is, features are ’diluted’ as they are forward propagated through
the model. Hence, later network layers receive less informative signals about the
input data, consequently making training less effective. In this work, we address
the problem of feature reuse by taking inspiration from an earlier work which
employed residual learning for alleviating the problem of feature reuse. We propose
a modification of residual learning for training very deep networks to realize
improved generalization performance; for this, we allow stochastic shortcut connections
of identity mappings from the input to hidden layers.We perform extensive
experiments using the USPS and MNIST datasets. On the USPS dataset, we
achieve an error rate of 2.69% without employing any form of data augmentation
(or manipulation). On the MNIST dataset, we reach a comparable state-of-the-art
error rate of 0.52%. Particularly, these results are achieved without employing
any explicit regularization technique
vDNN: Virtualized Deep Neural Networks for Scalable, Memory-Efficient Neural Network Design
The most widely used machine learning frameworks require users to carefully
tune their memory usage so that the deep neural network (DNN) fits into the
DRAM capacity of a GPU. This restriction hampers a researcher's flexibility to
study different machine learning algorithms, forcing them to either use a less
desirable network architecture or parallelize the processing across multiple
GPUs. We propose a runtime memory manager that virtualizes the memory usage of
DNNs such that both GPU and CPU memory can simultaneously be utilized for
training larger DNNs. Our virtualized DNN (vDNN) reduces the average GPU memory
usage of AlexNet by up to 89%, OverFeat by 91%, and GoogLeNet by 95%, a
significant reduction in memory requirements of DNNs. Similar experiments on
VGG-16, one of the deepest and memory hungry DNNs to date, demonstrate the
memory-efficiency of our proposal. vDNN enables VGG-16 with batch size 256
(requiring 28 GB of memory) to be trained on a single NVIDIA Titan X GPU card
containing 12 GB of memory, with 18% performance loss compared to a
hypothetical, oracular GPU with enough memory to hold the entire DNN.Comment: Published as a conference paper at the 49th IEEE/ACM International
Symposium on Microarchitecture (MICRO-49), 201
- …