6,880 research outputs found
Efficient Deep Feature Learning and Extraction via StochasticNets
Deep neural networks are a powerful tool for feature learning and extraction
given their ability to model high-level abstractions in highly complex data.
One area worth exploring in feature learning and extraction using deep neural
networks is efficient neural connectivity formation for faster feature learning
and extraction. Motivated by findings of stochastic synaptic connectivity
formation in the brain as well as the brain's uncanny ability to efficiently
represent information, we propose the efficient learning and extraction of
features via StochasticNets, where sparsely-connected deep neural networks can
be formed via stochastic connectivity between neurons. To evaluate the
feasibility of such a deep neural network architecture for feature learning and
extraction, we train deep convolutional StochasticNets to learn abstract
features using the CIFAR-10 dataset, and extract the learned features from
images to perform classification on the SVHN and STL-10 datasets. Experimental
results show that features learned using deep convolutional StochasticNets,
with fewer neural connections than conventional deep convolutional neural
networks, can allow for better or comparable classification accuracy than
conventional deep neural networks: relative test error decrease of ~4.5% for
classification on the STL-10 dataset and ~1% for classification on the SVHN
dataset. Furthermore, it was shown that the deep features extracted using deep
convolutional StochasticNets can provide comparable classification accuracy
even when only 10% of the training data is used for feature learning. Finally,
it was also shown that significant gains in feature extraction speed can be
achieved in embedded applications using StochasticNets. As such, StochasticNets
allow for faster feature learning and extraction performance while facilitate
for better or comparable accuracy performances.Comment: 10 pages. arXiv admin note: substantial text overlap with
arXiv:1508.0546
Genomics and proteomics: a signal processor's tour
The theory and methods of signal processing are becoming increasingly important in molecular biology. Digital filtering techniques, transform domain methods, and Markov models have played important roles in gene identification, biological sequence analysis, and alignment. This paper contains a brief review of molecular biology, followed by a review of the applications of signal processing theory. This includes the problem of gene finding using digital filtering, and the use of transform domain methods in the study of protein binding spots. The relatively new topic of noncoding genes, and the associated problem of identifying ncRNA buried in DNA sequences are also described. This includes a discussion of hidden Markov models and context free grammars. Several new directions in genomic signal processing are briefly outlined in the end
Stochasticity from function -- why the Bayesian brain may need no noise
An increasing body of evidence suggests that the trial-to-trial variability
of spiking activity in the brain is not mere noise, but rather the reflection
of a sampling-based encoding scheme for probabilistic computing. Since the
precise statistical properties of neural activity are important in this
context, many models assume an ad-hoc source of well-behaved, explicit noise,
either on the input or on the output side of single neuron dynamics, most often
assuming an independent Poisson process in either case. However, these
assumptions are somewhat problematic: neighboring neurons tend to share
receptive fields, rendering both their input and their output correlated; at
the same time, neurons are known to behave largely deterministically, as a
function of their membrane potential and conductance. We suggest that spiking
neural networks may, in fact, have no need for noise to perform sampling-based
Bayesian inference. We study analytically the effect of auto- and
cross-correlations in functionally Bayesian spiking networks and demonstrate
how their effect translates to synaptic interaction strengths, rendering them
controllable through synaptic plasticity. This allows even small ensembles of
interconnected deterministic spiking networks to simultaneously and
co-dependently shape their output activity through learning, enabling them to
perform complex Bayesian computation without any need for noise, which we
demonstrate in silico, both in classical simulation and in neuromorphic
emulation. These results close a gap between the abstract models and the
biology of functionally Bayesian spiking networks, effectively reducing the
architectural constraints imposed on physical neural substrates required to
perform probabilistic computing, be they biological or artificial
Analysis of DC microgrids as stochastic hybrid systems
A modeling framework for dc microgrids and distribution systems based on the dual active bridge (DAB) topology is presented. The purpose of this framework is to accurately characterize dynamic behavior of multi-converter systems as a function of exogenous load and source inputs. The base model is derived for deterministic inputs and then extended for the case of stochastic load behavior. At the core of the modeling framework is a large-signal DAB model that accurately describes the dynamics of both ac and dc state variables. This model addresses limitations of existing DAB converter models, which are not suitable for system-level analysis due to inaccuracy and poor upward scalability. The converter model acts as a fundamental building block in a general procedure for constructing models of multi-converter systems. System-level model construction is only possible due to structural properties of the converter model that mitigate prohibitive increases in size and complexity.
To characterize the impact of randomness in practical loads, stochastic load descriptions are included in the deterministic dynamic model. The combined behavior of distributed loads is represented by a continuous-time stochastic process. Models that govern this load process are generated using a new modeling procedure, which builds incrementally from individual device-level representations. To merge the stochastic load process and deterministic dynamic models, the microgrid is modeled as a stochastic hybrid system. The stochastic hybrid model predicts the evolution of moments of dynamic state variables as a function of load model parameters. Moments of dynamic states provide useful approximations of typical system operating conditions over time. Applications of the deterministic models include system stability analysis and computationally efficient time-domain simulation. The stochastic hybrid models provide a framework for performance assessment and optimization --Abstract, page iv
Training LSTM Networks with Resistive Cross-Point Devices
In our previous work we have shown that resistive cross point devices, so
called Resistive Processing Unit (RPU) devices, can provide significant power
and speed benefits when training deep fully connected networks as well as
convolutional neural networks. In this work, we further extend the RPU concept
for training recurrent neural networks (RNNs) namely LSTMs. We show that the
mapping of recurrent layers is very similar to the mapping of fully connected
layers and therefore the RPU concept can potentially provide large acceleration
factors for RNNs as well. In addition, we study the effect of various device
imperfections and system parameters on training performance. Symmetry of
updates becomes even more crucial for RNNs; already a few percent asymmetry
results in an increase in the test error compared to the ideal case trained
with floating point numbers. Furthermore, the input signal resolution to device
arrays needs to be at least 7 bits for successful training. However, we show
that a stochastic rounding scheme can reduce the input signal resolution back
to 5 bits. Further, we find that RPU device variations and hardware noise are
enough to mitigate overfitting, so that there is less need for using dropout.
We note that the models trained here are roughly 1500 times larger than the
fully connected network trained on MNIST dataset in terms of the total number
of multiplication and summation operations performed per epoch. Thus, here we
attempt to study the validity of the RPU approach for large scale networks.Comment: 17 pages, 5 figure
- …