27,353 research outputs found
Edge-Host Partitioning of Deep Neural Networks with Feature Space Encoding for Resource-Constrained Internet-of-Things Platforms
This paper introduces partitioning an inference task of a deep neural network
between an edge and a host platform in the IoT environment. We present a DNN as
an encoding pipeline, and propose to transmit the output feature space of an
intermediate layer to the host. The lossless or lossy encoding of the feature
space is proposed to enhance the maximum input rate supported by the edge
platform and/or reduce the energy of the edge platform. Simulation results show
that partitioning a DNN at the end of convolutional (feature extraction) layers
coupled with feature space encoding enables significant improvement in the
energy-efficiency and throughput over the baseline configurations that perform
the entire inference at the edge or at the host
Prediction of Aerodynamic Flow Fields Using Convolutional Neural Networks
An approximation model based on convolutional neural networks (CNNs) is
proposed for flow field predictions. The CNN is used to predict the velocity
and pressure field in unseen flow conditions and geometries given the pixelated
shape of the object. In particular, we consider Reynolds Averaged Navier-Stokes
(RANS) flow solutions over airfoil shapes. The CNN can automatically detect
essential features with minimal human supervision and shown to effectively
estimate the velocity and pressure field orders of magnitude faster than the
RANS solver, making it possible to study the impact of the airfoil shape and
operating conditions on the aerodynamic forces and the flow field in near-real
time. The use of specific convolution operations, parameter sharing, and
robustness to noise are shown to enhance the predictive capabilities of CNN. We
explore the network architecture and its effectiveness in predicting the flow
field for different airfoil shapes, angles of attack, and Reynolds numbers
Unsupervised Deep Feature Extraction for Remote Sensing Image Classification
This paper introduces the use of single layer and deep convolutional networks
for remote sensing data analysis. Direct application to multi- and
hyper-spectral imagery of supervised (shallow or deep) convolutional networks
is very challenging given the high input data dimensionality and the relatively
small amount of available labeled data. Therefore, we propose the use of greedy
layer-wise unsupervised pre-training coupled with a highly efficient algorithm
for unsupervised learning of sparse features. The algorithm is rooted on sparse
representations and enforces both population and lifetime sparsity of the
extracted features, simultaneously. We successfully illustrate the expressive
power of the extracted representations in several scenarios: classification of
aerial scenes, as well as land-use classification in very high resolution
(VHR), or land-cover classification from multi- and hyper-spectral images. The
proposed algorithm clearly outperforms standard Principal Component Analysis
(PCA) and its kernel counterpart (kPCA), as well as current state-of-the-art
algorithms of aerial classification, while being extremely computationally
efficient at learning representations of data. Results show that single layer
convolutional networks can extract powerful discriminative features only when
the receptive field accounts for neighboring pixels, and are preferred when the
classification requires high resolution and detailed results. However, deep
architectures significantly outperform single layers variants, capturing
increasing levels of abstraction and complexity throughout the feature
hierarchy
Deep convolutional encoder-decoder networks for uncertainty quantification of dynamic multiphase flow in heterogeneous media
Surrogate strategies are used widely for uncertainty quantification of
groundwater models in order to improve computational efficiency. However, their
application to dynamic multiphase flow problems is hindered by the curse of
dimensionality, the saturation discontinuity due to capillarity effects, and
the time-dependence of the multi-output responses. In this paper, we propose a
deep convolutional encoder-decoder neural network methodology to tackle these
issues. The surrogate modeling task is transformed to an image-to-image
regression strategy. This approach extracts high-level coarse features from the
high-dimensional input permeability images using an encoder, and then refines
the coarse features to provide the output pressure/saturation images through a
decoder. A training strategy combining a regression loss and a segmentation
loss is proposed in order to better approximate the discontinuous saturation
field. To characterize the high-dimensional time-dependent outputs of the
dynamic system, time is treated as an additional input to the network that is
trained using pairs of input realizations and of the corresponding system
outputs at a limited number of time instances. The proposed method is evaluated
using a geological carbon storage process-based multiphase flow model with a
2500-dimensional stochastic permeability field. With a relatively small number
of training data, the surrogate model is capable of accurately characterizing
the spatio-temporal evolution of the pressure and discontinuous CO2 saturation
fields and can be used efficiently to compute the statistics of the system
responses.Comment: 30 pages, 21 figures, submitted to Water Resources Researc
Whetstone: A Method for Training Deep Artificial Neural Networks for Binary Communication
This paper presents a new technique for training networks for low-precision
communication. Targeting minimal communication between nodes not only enables
the use of emerging spiking neuromorphic platforms, but may additionally
streamline processing conventionally. Low-power and embedded neuromorphic
processors potentially offer dramatic performance-per-Watt improvements over
traditional von Neumann processors, however programming these brain-inspired
platforms generally requires platform-specific expertise which limits their
applicability. To date, the majority of artificial neural networks have not
operated using discrete spike-like communication.
We present a method for training deep spiking neural networks using an
iterative modification of the backpropagation optimization algorithm. This
method, which we call Whetstone, effectively and reliably configures a network
for a spiking hardware target with little, if any, loss in performance.
Whetstone networks use single time step binary communication and do not require
a rate code or other spike-based coding scheme, thus producing networks
comparable in timing and size to conventional ANNs, albeit with binarized
communication. We demonstrate Whetstone on a number of image classification
networks, describing how the sharpening process interacts with different
training optimizers and changes the distribution of activity within the
network. We further note that Whetstone is compatible with several
non-classification neural network applications, such as autoencoders and
semantic segmentation. Whetstone is widely extendable and currently implemented
using custom activation functions within the Keras wrapper to the popular
TensorFlow machine learning framework
Deep and Wide Multiscale Recursive Networks for Robust Image Labeling
Feedforward multilayer networks trained by supervised learning have recently
demonstrated state of the art performance on image labeling problems such as
boundary prediction and scene parsing. As even very low error rates can limit
practical usage of such systems, methods that perform closer to human accuracy
remain desirable. In this work, we propose a new type of network with the
following properties that address what we hypothesize to be limiting aspects of
existing methods: (1) a `wide' structure with thousands of features, (2) a
large field of view, (3) recursive iterations that exploit statistical
dependencies in label space, and (4) a parallelizable architecture that can be
trained in a fraction of the time compared to benchmark multilayer
convolutional networks. For the specific image labeling problem of boundary
prediction, we also introduce a novel example weighting algorithm that improves
segmentation accuracy. Experiments in the challenging domain of connectomic
reconstruction of neural circuity from 3d electron microscopy data show that
these "Deep And Wide Multiscale Recursive" (DAWMR) networks lead to new levels
of image labeling performance. The highest performing architecture has twelve
layers, interwoven supervised and unsupervised stages, and uses an input field
of view of 157,464 voxels () to make a prediction at each image location.
We present an associated open source software package that enables the simple
and flexible creation of DAWMR networks
CodeX: Bit-Flexible Encoding for Streaming-based FPGA Acceleration of DNNs
This paper proposes CodeX, an end-to-end framework that facilitates encoding,
bitwidth customization, fine-tuning, and implementation of neural networks on
FPGA platforms. CodeX incorporates nonlinear encoding to the computation flow
of neural networks to save memory. The encoded features demand significantly
lower storage compared to the raw full-precision activation values; therefore,
the execution flow of CodeX hardware engine is completely performed within the
FPGA using on-chip streaming buffers with no access to the off-chip DRAM. We
further propose a fully-automated algorithm inspired by reinforcement learning
which determines the customized encoding bitwidth across network layers. CodeX
full-stack framework comprises of a compiler which takes a high-level Python
description of an arbitrary neural network architecture. The compiler then
instantiates the corresponding elements from CodeX Hardware library for FPGA
implementation. Proof-of-concept evaluations on MNIST, SVHN, and CIFAR-10
datasets demonstrate an average of 4.65x throughput improvement compared to
stand-alone weight encoding. We further compare CodeX with six existing
full-precision DNN accelerators on ImageNet, showing an average of 3.6x and
2.54x improvement in throughput and performance-per-watt, respectively
Automatic Model Selection for Neural Networks
Neural networks and deep learning are changing the way that artificial
intelligence is being done. Efficiently choosing a suitable network
architecture and fine-tune its hyper-parameters for a specific dataset is a
time-consuming task given the staggering number of possible alternatives. In
this paper, we address the problem of model selection by means of a fully
automated framework for efficiently selecting a neural network model for a
given task: classification or regression. The algorithm, named Automatic Model
Selection, is a modified micro-genetic algorithm that automatically and
efficiently finds the most suitable neural network model for a given dataset.
The main contributions of this method are a simple list based encoding for
neural networks as genotypes in an evolutionary algorithm, new crossover, and
mutation operators, the introduction of a fitness function that considers both,
the accuracy of the model and its complexity and a method to measure the
similarity between two neural networks. AMS is evaluated on two different
datasets. By comparing some models obtained with AMS to state-of-the-art models
for each dataset we show that AMS can automatically find efficient neural
network models. Furthermore, AMS is computationally efficient and can make use
of distributed computing paradigms to further boost its performance.Comment: 31 pages, 6 figures. Preprint Submitted to Elsevier Neural Network
NeuNetS: An Automated Synthesis Engine for Neural Network Design
Application of neural networks to a vast variety of practical applications is
transforming the way AI is applied in practice. Pre-trained neural network
models available through APIs or capability to custom train pre-built neural
network architectures with customer data has made the consumption of AI by
developers much simpler and resulted in broad adoption of these complex AI
models. While prebuilt network models exist for certain scenarios, to try and
meet the constraints that are unique to each application, AI teams need to
think about developing custom neural network architectures that can meet the
tradeoff between accuracy and memory footprint to achieve the tight constraints
of their unique use-cases. However, only a small proportion of data science
teams have the skills and experience needed to create a neural network from
scratch, and the demand far exceeds the supply. In this paper, we present
NeuNetS : An automated Neural Network Synthesis engine for custom neural
network design that is available as part of IBM's AI OpenScale's product.
NeuNetS is available for both Text and Image domains and can build neural
networks for specific tasks in a fraction of the time it takes today with human
effort, and with accuracy similar to that of human-designed AI models.Comment: 14 pages, 12 figures. arXiv admin note: text overlap with
arXiv:1806.0025
Graph Convolution: A High-Order and Adaptive Approach
In this paper, we presented a novel convolutional neural network framework
for graph modeling, with the introduction of two new modules specially designed
for graph-structured data: the -th order convolution operator and the
adaptive filtering module. Importantly, our framework of High-order and
Adaptive Graph Convolutional Network (HA-GCN) is a general-purposed
architecture that fits various applications on both node and graph centrics, as
well as graph generative models. We conducted extensive experiments on
demonstrating the advantages of our framework. Particularly, our HA-GCN
outperforms the state-of-the-art models on node classification and molecule
property prediction tasks. It also generates 32% more real molecules on the
molecule generation task, both of which will significantly benefit real-world
applications such as material design and drug screening
- …