42,855 research outputs found
Convolution with even-sized kernels and symmetric padding
Compact convolutional neural networks gain efficiency mainly through
depthwise convolutions, expanded channels and complex topologies, which
contrarily aggravate the training process. Besides, 3x3 kernels dominate the
spatial representation in these models, whereas even-sized kernels (2x2, 4x4)
are rarely adopted. In this work, we quantify the shift problem occurs in
even-sized kernel convolutions by an information erosion hypothesis, and
eliminate it by proposing symmetric padding on four sides of the feature maps
(C2sp, C4sp). Symmetric padding releases the generalization capabilities of
even-sized kernels at little computational cost, making them outperform 3x3
kernels in image classification and generation tasks. Moreover, C2sp obtains
comparable accuracy to emerging compact models with much less memory and time
consumption during training. Symmetric padding coupled with even-sized
convolutions can be neatly implemented into existing frameworks, providing
effective elements for architecture designs, especially on online and continual
learning occasions where training efforts are emphasized.Comment: 12 pages, 4 figures, 4 table
A Computationally Efficient Neural Network Invariant to the Action of Symmetry Subgroups
We introduce a method to design a computationally efficient -invariant
neural network that approximates functions invariant to the action of a given
permutation subgroup of the symmetric group on input data. The key
element of the proposed network architecture is a new -invariant
transformation module, which produces a -invariant latent representation of
the input data. This latent representation is then processed with a multi-layer
perceptron in the network. We prove the universality of the proposed
architecture, discuss its properties and highlight its computational and memory
efficiency. Theoretical considerations are supported by numerical experiments
involving different network configurations, which demonstrate the effectiveness
and strong generalization properties of the proposed method in comparison to
other -invariant neural networks
On the Expressive Power of Deep Learning: A Tensor Analysis
It has long been conjectured that hypotheses spaces suitable for data that is
compositional in nature, such as text or images, may be more efficiently
represented with deep hierarchical networks than with shallow ones. Despite the
vast empirical evidence supporting this belief, theoretical justifications to
date are limited. In particular, they do not account for the locality, sharing
and pooling constructs of convolutional networks, the most successful deep
learning architecture to date. In this work we derive a deep network
architecture based on arithmetic circuits that inherently employs locality,
sharing and pooling. An equivalence between the networks and hierarchical
tensor factorizations is established. We show that a shallow network
corresponds to CP (rank-1) decomposition, whereas a deep network corresponds to
Hierarchical Tucker decomposition. Using tools from measure theory and matrix
algebra, we prove that besides a negligible set, all functions that can be
implemented by a deep network of polynomial size, require exponential size in
order to be realized (or even approximated) by a shallow network. Since
log-space computation transforms our networks into SimNets, the result applies
directly to a deep learning architecture demonstrating promising empirical
performance. The construction and theory developed in this paper shed new light
on various practices and ideas employed by the deep learning community
S-Net: A Scalable Convolutional Neural Network for JPEG Compression Artifact Reduction
Recent studies have used deep residual convolutional neural networks (CNNs)
for JPEG compression artifact reduction. This study proposes a scalable CNN
called S-Net. Our approach effectively adjusts the network scale dynamically in
a multitask system for real-time operation with little performance loss. It
offers a simple and direct technique to evaluate the performance gains obtained
with increasing network depth, and it is helpful for removing redundant network
layers to maximize the network efficiency. We implement our architecture using
the Keras framework with the TensorFlow backend on an NVIDIA K80 GPU server. We
train our models on the DIV2K dataset and evaluate their performance on public
benchmark datasets. To validate the generality and universality of the proposed
method, we created and utilized a new dataset, called WIN143, for
over-processed images evaluation. Experimental results indicate that our
proposed approach outperforms other CNN-based methods and achieves
state-of-the-art performance.Comment: accepted by Journal of Electronic Imagin
PZnet: Efficient 3D ConvNet Inference on Manycore CPUs
Convolutional nets have been shown to achieve state-of-the-art accuracy in
many biomedical image analysis tasks. Many tasks within biomedical analysis
domain involve analyzing volumetric (3D) data acquired by CT, MRI and
Microscopy acquisition methods. To deploy convolutional nets in practical
working systems, it is important to solve the efficient inference problem.
Namely, one should be able to apply an already-trained convolutional network to
many large images using limited computational resources. In this paper we
present PZnet, a CPU-only engine that can be used to perform inference for a
variety of 3D convolutional net architectures. PZNet outperforms MKL-based CPU
implementations of PyTorch and Tensorflow by more than 3.5x for the popular
U-net architecture. Moreover, for 3D convolutions with low featuremap numbers,
cloud CPU inference with PZnet outperfroms cloud GPU inference in terms of cost
efficiency
Bootstrapping Deep Neural Networks from Approximate Image Processing Pipelines
Complex image processing and computer vision systems often consist of a
processing pipeline of functional modules. We intend to replace parts or all of
a target pipeline with deep neural networks to achieve benefits such as
increased accuracy or reduced computational requirement. To acquire a large
amount of labeled data necessary to train the deep neural network, we propose a
workflow that leverages the target pipeline to create a significantly larger
labeled training set automatically, without prior domain knowledge of the
target pipeline. We show experimentally that despite the noise introduced by
automated labeling and only using a very small initially labeled data set, the
trained deep neural networks can achieve similar or even better performance
than the components they replace, while in some cases also reducing
computational requirements.Comment: 6 pages, 5 figure
Solving Many-Electron Schr\"odinger Equation Using Deep Neural Networks
We introduce a new family of trial wave-functions based on deep neural
networks to solve the many-electron Schr\"odinger equation. The Pauli exclusion
principle is dealt with explicitly to ensure that the trial wave-functions are
physical. The optimal trial wave-function is obtained through variational Monte
Carlo and the computational cost scales quadratically with the number of
electrons. The algorithm does not make use of any prior knowledge such as
atomic orbitals. Yet it is able to represent accurately the ground-states of
the tested systems, including He, H2, Be, B, LiH, and a chain of 10 hydrogen
atoms. This opens up new possibilities for solving large-scale many-electron
Schr\"odinger equation
A theoretical basis for efficient computations with noisy spiking neurons
Network of neurons in the brain apply - unlike processors in our current
generation of computer hardware - an event-based processing strategy, where
short pulses (spikes) are emitted sparsely by neurons to signal the occurrence
of an event at a particular point in time. Such spike-based computations
promise to be substantially more power-efficient than traditional clocked
processing schemes. However it turned out to be surprisingly difficult to
design networks of spiking neurons that are able to carry out demanding
computations. We present here a new theoretical framework for organizing
computations of networks of spiking neurons. In particular, we show that a
suitable design enables them to solve hard constraint satisfaction problems
from the domains of planning - optimization and verification - logical
inference. The underlying design principles employ noise as a computational
resource. Nevertheless the timing of spikes (rather than just spike rates)
plays an essential role in the resulting computations. Furthermore, one can
demonstrate for the Traveling Salesman Problem a surprising computational
advantage of networks of spiking neurons compared with traditional artificial
neural networks and Gibbs sampling. The identification of such advantage has
been a well-known open problem.Comment: main paper: 21 pages, 5 figures supplemental paper: 11 pages, no
figure
Chaotic Simulated Annealing by A Neural Network Model with Transient Chaos
We propose a neural network model with transient chaos, or a transiently
chaotic neural network (TCNN) as an approximation method for combinatorial
optimization problem, by introducing transiently chaotic dynamics into neural
networks. Unlike conventional neural networks only with point attractors, the
proposed neural network has richer and more flexible dynamics, so that it can
be expected to have higher ability of searching for globally optimal or
near-optimal solutions. A significant property of this model is that the
chaotic neurodynamics is temporarily generated for searching and
self-organizing, and eventually vanishes with autonomous decreasing of a
bifurcation parameter corresponding to the "temperature" in usual annealing
process. Therefore, the neural network gradually approaches, through the
transient chaos, to dynamical structure similar to such conventional models as
the Hopfield neural network which converges to a stable equilibrium point.
Since the optimization process of the transiently chaotic neural network is
similar to simulated annealing, not in a stochastic way but in a
deterministically chaotic way, the new method is regarded as chaotic simulated
annealing (CSA). Fundamental characteristics of the transiently chaotic
neurodynamics are numerically investigated with examples of a single neuron
model and the Traveling Salesman Problem (TSP). Moreover, a maintenance
scheduling problem for generators in a practical power system is also analysed
to verify practical efficiency of this new method.Comment: the theoretical results related to this paper should be referred to
"Chaos and Asymptotical Stability in Discrete-time Neural Networks" by L.Chen
and K.Aihara, Physica D (in press). Journal ref.: Neural Networks, Vol.8,
No.6, pp.915-930, 199
Iterative methods for computing U-eigenvalues of non-symmetric complex tensors with application in quantum entanglement
The purpose of this paper is to study the problem of computing unitary
eigenvalues (U-eigenvalues) of non-symmetric complex tensors. By means of
symmetric embedding of complex tensors, the relationship between U-eigenpairs
of a non-symmetric complex tensor and unitary symmetric eigenpairs
(US-eigenpairs) of its symmetric embedding tensor is established. An algorithm
(Algorithm \ref{algo:1}) is given to compute the U-eigenvalues of non-symmetric
complex tensors by means of symmetric embedding. Another algorithm, Algorithm
\ref{algo:2}, is proposed to directly compute the U-eigenvalues of
non-symmetric complex tensors, without the aid of symmetric embedding. Finally,
a tensor version of the well-known Gauss-Seidel method is developed. Efficiency
of these three algorithms are compared by means of various numerical examples.
These algorithms are applied to compute the geometric measure of entanglement
of quantum multipartite non-symmetric pure states.Comment: 19 pages; submitted for publication; comments are welcom
- …