17,906 research outputs found
Quality Control of Neuron Reconstruction Based on Deep Learning
Neuron reconstruction is essential to generate exquisite neuron connectivity
map for understanding brain function. Despite the significant amount of effect
that has been made on automatic reconstruction methods, manual tracing by
well-trained human annotators is still necessary. To ensure the quality of
reconstructed neurons and provide guidance for annotators to improve their
efficiency, we propose a deep learning based quality control method for neuron
reconstruction in this paper. By formulating the quality control problem into a
binary classification task regarding each single point, the proposed approach
overcomes the technical difficulties resulting from the large image size and
complex neuron morphology. Not only it provides the evaluation of
reconstruction quality, but also can locate exactly where the wrong tracing
begins. This work presents one of the first comprehensive studies for
whole-brain scale quality control of neuron reconstructions. Experiments on
five-fold cross validation with a large dataset demonstrate that the proposed
approach can detect 74.7% errors with only 1.4% false alerts.Comment: 9 pages, 2 figure
Mapping Generative Models onto a Network of Digital Spiking Neurons
Stochastic neural networks such as Restricted Boltzmann Machines (RBMs) have
been successfully used in applications ranging from speech recognition to image
classification. Inference and learning in these algorithms use a Markov Chain
Monte Carlo procedure called Gibbs sampling, where a logistic function forms
the kernel of this sampler. On the other side of the spectrum, neuromorphic
systems have shown great promise for low-power and parallelized cognitive
computing, but lack well-suited applications and automation procedures. In this
work, we propose a systematic method for bridging the RBM algorithm and digital
neuromorphic systems, with a generative pattern completion task as proof of
concept. For this, we first propose a method of producing the Gibbs sampler
using bio-inspired digital noisy integrate-and-fire neurons. Next, we describe
the process of mapping generative RBMs trained offline onto the IBM TrueNorth
neurosynaptic processor -- a low-power digital neuromorphic VLSI substrate.
Mapping these algorithms onto neuromorphic hardware presents unique challenges
in network connectivity and weight and bias quantization, which, in turn,
require architectural and design strategies for the physical realization.
Generative performance metrics are analyzed to validate the neuromorphic
requirements and to best select the neuron parameters for the model. Lastly, we
describe a design automation procedure which achieves optimal resource usage,
accounting for the novel hardware adaptations. This work represents the first
implementation of generative RBM inference on a neuromorphic VLSI substrate.Comment: A similar version of this manuscript has been submitted to IEEE
TBioCAS for revision in October 201
PILAE: A Non-gradient Descent Learning Scheme for Deep Feedforward Neural Networks
In this work, a non-gradient descent learning scheme is proposed for deep
feedforward neural networks (DNN). As we known, autoencoder can be used as the
building blocks of the multi-layer perceptron (MLP) deep neural network. So,
the MLP will be taken as an example to illustrate the proposed scheme of
pseudoinverse learning algorithm for autoencoder (PILAE) training. The PILAE
with low rank approximation is a non-gradient based learning algorithm, and the
encoder weight matrix is set to be the low rank approximation of the
pseudoinverse of the input matrix, while the decoder weight matrix is
calculated by the pseudoinverse learning algorithm. It is worth to note that
only few network structure hyperparameters need to be tuned. Hence, the
proposed algorithm can be regarded as a quasi-automated training algorithm
which can be utilized in autonomous machine learning research field. The
experimental results show that the proposed learning scheme for DNN can achieve
better performance on considering the tradeoff between training efficiency and
classification accuracy.Comment: This work is our effort toward to realize AutoM
Revisit Lmser and its further development based on convolutional layers
Proposed in 1991, Least Mean Square Error Reconstruction for self-organizing
network, shortly Lmser, was a further development of the traditional
auto-encoder (AE) by folding the architecture with respect to the central
coding layer and thus leading to the features of symmetric weights and neurons,
as well as jointly supervised and unsupervised learning. However, its
advantages were only demonstrated in a one-hidden-layer implementation due to
the lack of computing resources and big data at that time. In this paper, we
revisit Lmser from the perspective of deep learning, develop Lmser network
based on multiple convolutional layers, which is more suitable for
image-related tasks, and confirm several Lmser functions with preliminary
demonstrations on image recognition, reconstruction, association recall, and so
on. Experiments demonstrate that Lmser indeed works as indicated in the
original paper, and it has promising performance in various applications
Deep Learning with the Random Neural Network and its Applications
The random neural network (RNN) is a mathematical model for an "integrate and
fire" spiking network that closely resembles the stochastic behaviour of
neurons in mammalian brains. Since its proposal in 1989, there have been
numerous investigations into the RNN's applications and learning algorithms.
Deep learning (DL) has achieved great success in machine learning. Recently,
the properties of the RNN for DL have been investigated, in order to combine
their power. Recent results demonstrate that the gap between RNNs and DL can be
bridged and the DL tools based on the RNN are faster and can potentially be
used with less energy expenditure than existing methods.Comment: 23 pages, 19 figure
Revisit Fuzzy Neural Network: Demystifying Batch Normalization and ReLU with Generalized Hamming Network
We revisit fuzzy neural network with a cornerstone notion of generalized
hamming distance, which provides a novel and theoretically justified framework
to re-interpret many useful neural network techniques in terms of fuzzy logic.
In particular, we conjecture and empirically illustrate that, the celebrated
batch normalization (BN) technique actually adapts the normalized bias such
that it approximates the rightful bias induced by the generalized hamming
distance. Once the due bias is enforced analytically, neither the optimization
of bias terms nor the sophisticated batch normalization is needed. Also in the
light of generalized hamming distance, the popular rectified linear units
(ReLU) can be treated as setting a minimal hamming distance threshold between
network inputs and weights. This thresholding scheme, on the one hand, can be
improved by introducing double thresholding on both extremes of neuron outputs.
On the other hand, ReLUs turn out to be non-essential and can be removed from
networks trained for simple tasks like MNIST classification. The proposed
generalized hamming network (GHN) as such not only lends itself to rigorous
analysis and interpretation within the fuzzy logic theory but also demonstrates
fast learning speed, well-controlled behaviour and state-of-the-art
performances on a variety of learning tasks.Comment: 10 pages, 5 figures, NIPS 201
A Gentle Introduction to Deep Learning in Medical Image Processing
This paper tries to give a gentle introduction to deep learning in medical
image processing, proceeding from theoretical foundations to applications. We
first discuss general reasons for the popularity of deep learning, including
several major breakthroughs in computer science. Next, we start reviewing the
fundamental basics of the perceptron and neural networks, along with some
fundamental theory that is often omitted. Doing so allows us to understand the
reasons for the rise of deep learning in many application domains. Obviously
medical image processing is one of these areas which has been largely affected
by this rapid progress, in particular in image detection and recognition, image
segmentation, image registration, and computer-aided diagnosis. There are also
recent trends in physical simulation, modelling, and reconstruction that have
led to astonishing results. Yet, some of these approaches neglect prior
knowledge and hence bear the risk of producing implausible results. These
apparent weaknesses highlight current limitations of deep learning. However, we
also briefly discuss promising approaches that might be able to resolve these
problems in the future.Comment: Accepted by Journal of Medical Physics; Final Version after revie
Analysis of Extremely Obese Individuals Using Deep Learning Stacked Autoencoders and Genome-Wide Genetic Data
The aetiology of polygenic obesity is multifactorial, which indicates that
life-style and environmental factors may influence multiples genes to aggravate
this disorder. Several low-risk single nucleotide polymorphisms (SNPs) have
been associated with BMI. However, identified loci only explain a small
proportion of the variation ob-served for this phenotype. The linear nature of
genome wide association studies (GWAS) used to identify associations between
genetic variants and the phenotype have had limited success in explaining the
heritability variation of BMI and shown low predictive capacity in
classification studies. GWAS ignores the epistatic interactions that less
significant variants have on the phenotypic outcome. In this paper we utilise a
novel deep learning-based methodology to reduce the high dimensional space in
GWAS and find epistatic interactions between SNPs for classification purposes.
SNPs were filtered based on the effects associations have with BMI. Since
Bonferroni adjustment for multiple testing is highly conservative, an important
proportion of SNPs involved in SNP-SNP interactions are ignored. Therefore,
only SNPs with p-values < 1x10-2 were considered for subsequent epistasis
analysis using stacked auto encoders (SAE). This allows the nonlinearity
present in SNP-SNP interactions to be discovered through progressively smaller
hidden layer units and to initialise a multi-layer feedforward artificial
neural network (ANN) classifier. The classifier is fine-tuned to classify
extremely obese and non-obese individuals. The best results were obtained with
2000 compressed units (SE=0.949153, SP=0.933014, Gini=0.949936,
Lo-gloss=0.1956, AUC=0.97497 and MSE=0.054057). Using 50 compressed units it
was possible to achieve (SE=0.785311, SP=0.799043, Gini=0.703566,
Logloss=0.476864, AUC=0.85178 and MSE=0.156315).Comment: 13 pages, 4 figures, 13 equations, 2 tables, conferenc
Stochastic Bottleneck: Rateless Auto-Encoder for Flexible Dimensionality Reduction
We propose a new concept of rateless auto-encoders (RL-AEs) that enable a
flexible latent dimensionality, which can be seamlessly adjusted for varying
distortion and dimensionality requirements. In the proposed RL-AEs, instead of
a deterministic bottleneck architecture, we use an over-complete representation
that is stochastically regularized with weighted dropouts, in a manner
analogous to sparse AE (SAE). Unlike SAEs, our RL-AEs employ monotonically
increasing dropout rates across the latent representation nodes such that the
latent variables become sorted by importance like in principal component
analysis (PCA). This is motivated by the rateless property of conventional PCA,
where the least important principal components can be discarded to realize
variable rate dimensionality reduction that gracefully degrades the distortion.
In contrast, since the latent variables of conventional AEs are equally
important for data reconstruction, they cannot be simply discarded to further
reduce the dimensionality after the AE model is trained. Our proposed
stochastic bottleneck framework enables seamless rate adaptation with high
reconstruction performance, without requiring predetermined latent
dimensionality at training. We experimentally demonstrate that the proposed
RL-AEs can achieve variable dimensionality reduction while achieving low
distortion compared to conventional AEs.Comment: 14 pages, 12 figures, ISIT 2020 accepte
A learning framework for winner-take-all networks with stochastic synapses
Many recent generative models make use of neural networks to transform the
probability distribution of a simple low-dimensional noise process into the
complex distribution of the data. This raises the question of whether
biological networks operate along similar principles to implement a
probabilistic model of the environment through transformations of intrinsic
noise processes. The intrinsic neural and synaptic noise processes in
biological networks, however, are quite different from the noise processes used
in current abstract generative networks. This, together with the discrete
nature of spikes and local circuit interactions among the neurons, raises
several difficulties when using recent generative modeling frameworks to train
biologically motivated models. In this paper, we show that a biologically
motivated model based on multi-layer winner-take-all (WTA) circuits and
stochastic synapses admits an approximate analytical description. This allows
us to use the proposed networks in a variational learning setting where
stochastic backpropagation is used to optimize a lower bound on the data log
likelihood, thereby learning a generative model of the data. We illustrate the
generality of the proposed networks and learning technique by using them in a
structured output prediction task, and in a semi-supervised learning task. Our
results extend the domain of application of modern stochastic network
architectures to networks where synaptic transmission failure is the principal
noise mechanism
- …