26 research outputs found
Zero-bias autoencoders and the benefits of co-adapting features
Regularized training of an autoencoder typically results in hidden unit
biases that take on large negative values. We show that negative biases are a
natural result of using a hidden layer whose responsibility is to both
represent the input data and act as a selection mechanism that ensures sparsity
of the representation. We then show that negative biases impede the learning of
data distributions whose intrinsic dimensionality is high. We also propose a
new activation function that decouples the two roles of the hidden layer and
that allows us to learn representations on data with very high intrinsic
dimensionality, where standard autoencoders typically fail. Since the decoupled
activation function acts like an implicit regularizer, the model can be trained
by minimizing the reconstruction error of training data, without requiring any
additional regularization
Fiber Orientation Estimation Guided by a Deep Network
Diffusion magnetic resonance imaging (dMRI) is currently the only tool for
noninvasively imaging the brain's white matter tracts. The fiber orientation
(FO) is a key feature computed from dMRI for fiber tract reconstruction.
Because the number of FOs in a voxel is usually small, dictionary-based sparse
reconstruction has been used to estimate FOs with a relatively small number of
diffusion gradients. However, accurate FO estimation in regions with complex FO
configurations in the presence of noise can still be challenging. In this work
we explore the use of a deep network for FO estimation in a dictionary-based
framework and propose an algorithm named Fiber Orientation Reconstruction
guided by a Deep Network (FORDN). FORDN consists of two steps. First, we use a
smaller dictionary encoding coarse basis FOs to represent the diffusion
signals. To estimate the mixture fractions of the dictionary atoms (and thus
coarse FOs), a deep network is designed specifically for solving the sparse
reconstruction problem. Here, the smaller dictionary is used to reduce the
computational cost of training. Second, the coarse FOs inform the final FO
estimation, where a larger dictionary encoding dense basis FOs is used and a
weighted l1-norm regularized least squares problem is solved to encourage FOs
that are consistent with the network output. FORDN was evaluated and compared
with state-of-the-art algorithms that estimate FOs using sparse reconstruction
on simulated and real dMRI data, and the results demonstrate the benefit of
using a deep network for FO estimation.Comment: A shorter version is accepted by MICCAI 201
Learning Combinations of Activation Functions
In the last decade, an active area of research has been devoted to design
novel activation functions that are able to help deep neural networks to
converge, obtaining better performance. The training procedure of these
architectures usually involves optimization of the weights of their layers
only, while non-linearities are generally pre-specified and their (possible)
parameters are usually considered as hyper-parameters to be tuned manually. In
this paper, we introduce two approaches to automatically learn different
combinations of base activation functions (such as the identity function, ReLU,
and tanh) during the training phase. We present a thorough comparison of our
novel approaches with well-known architectures (such as LeNet-5, AlexNet, and
ResNet-56) on three standard datasets (Fashion-MNIST, CIFAR-10, and
ILSVRC-2012), showing substantial improvements in the overall performance, such
as an increase in the top-1 accuracy for AlexNet on ILSVRC-2012 of 3.01
percentage points.Comment: 6 pages, 3 figures. Published as a conference paper at ICPR 2018.
Code:
https://bitbucket.org/francux/learning_combinations_of_activation_function