Accelerating Stochastic Random Projection Neural Networks

Artificial Neural Network (ANN), a computational model based on the biological neural networks, has a recent resurgence in machine intelligence with breakthrough results in pattern recognition, speech recognition, and mapping. This has led to a growing interest in designing dedicated hardware substrates for ANNs with a goal of achieving energy efficiency, high network connectivity and better computational capabilities that are typically not optimized in software ANN stack. Using stochastic computing is a natural choice to reduce the total system energy, where a signal is expressed through the statistical distribution of the logical values as a random bit stream. Generally, the accuracy of these systems is correlated with the stochastic bit stream length and requires long compute times. In this work, a framework is proposed to accelerate the long compute times in stochastic ANNs. A GPU acceleration framework has been developed to validate two random projection networks to test the efficacy of these networks prior to custom hardware design. The networks are stochastic extreme learning machine, a supervised feed-forward neural network and stochastic echo state network, a recurrent neural network with online learning. The framework also provisions identifying optimal values for various network parameters like learning rate, number of hidden layers and stochastic number length. The proposed stochastic extreme learning machine design is validated for two standardized datasets, MNIST dataset and orthopedic dataset. The proposed stochastic echo state network is validated on the time series EEG dataset. The CPU models were developed for each of these networks to calculate the relative performance boost. The design knobs for performance boost include stochastic bit stream generation, activation function, reservoir layer and training unit of the networks. Proposed stochastic extreme learning machine and stochastic echo state network achieved a performance boost of 60.61x for Orthopedic dataset and 42.03x for EEG dataset with 2^12 bit stream length when tested on an Nvidia GeForce1050 Ti

Enhancing Energy Minimization Framework for Scene Text Recognition with Top-Down Cues

Recognizing scene text is a challenging problem, even more so than the recognition of scanned documents. This problem has gained significant attention from the computer vision community in recent years, and several methods based on energy minimization frameworks and deep learning approaches have been proposed. In this work, we focus on the energy minimization framework and propose a model that exploits both bottom-up and top-down cues for recognizing cropped words extracted from street images. The bottom-up cues are derived from individual character detections from an image. We build a conditional random field model on these detections to jointly model the strength of the detections and the interactions between them. These interactions are top-down cues obtained from a lexicon-based prior, i.e., language statistics. The optimal word represented by the text image is obtained by minimizing the energy function corresponding to the random field model. We evaluate our proposed algorithm extensively on a number of cropped scene text benchmark datasets, namely Street View Text, ICDAR 2003, 2011 and 2013 datasets, and IIIT 5K-word, and show better performance than comparable methods. We perform a rigorous analysis of all the steps in our approach and analyze the results. We also show that state-of-the-art convolutional neural network features can be integrated in our framework to further improve the recognition performance

Evaluation of process planning in manufacturing by a neural network based on an energy definition of hopfield nets

During the planning stages of new factories for the Body-In-White assembly, the processes used per production system need to be defined. Each production system uses a specific combination of processes, with each process belonging to a main process group. The combination of the processes and groups is subject to restrictions. Since the amount of possible combinations is too large to individually check for restrictions, we propose a Neural Network using an energy measurement derived from Hopfield networks. The proposed network memorizes former correct combinations and provides a recommendation score on how likely a new planned configuration is. Since processes can be paired with processes from their own group or with themselves, the Neural Network is modified to allow loops for joining vertices with themselves. This modification is achieved by adjusting the energy function of Hopfield networks to measure the activation of the combinations of clusters, meaning the edges, and not the activation of vertices during the training phase. We implemented the network for the process planning of factories of a leading European automotive manufacturer, and the results using correct, incorrect, and random process combinations indicate a strong capability of detecting anomalous process combinations

Image segmentation using a neural network

An object extraction problem based on the Gibbs Random Field model is discussed. The Maximum a'posteriori probability (MAP) estimate of a scene based on a noise-corrupted realization is found to be computationally exponential in nature. A neural network, which is a modified version of that of Hopfield, is suggested for solving the problem. A single neuron is assigned to every pixel. Each neuron is supposed to be connected only to all of its nearest neighbours. The energy function of the network is designed in such a way that its minimum value corresponds to the MAP estimate of the scene. The dynamics of the network are described. A possible hardware realization of a neuron is also suggested. The technique is implemented on a set of noisy images and found to be highly robust and immune to noise

A Generative Neural Network for Discovering Near Optimaldynamic Inductive Power Transfer Systems

An urgent need is to electrify transportation to lower carbon emissions into the atmosphere. Wireless charging makes electrical vehicles (EVs) more convenient and cheaper because energy is transferred to the vehicle without the need to plug it in. Dynamic wireless charging is particularly interesting, where the vehicle does not need to stop to receive the energy. This technology requires the EV and the roadway to include coils of wire, where the roadway coil is energized as the vehicle passes over it to induce an electrical current in the EV coil through electromagnetic induction. However, the problem of designing the two coils (EV and road) is complex due to the many configurations possible, the need to maximize power transfer, and the need to minimize stray, possibly dangerous, electromagnetic fields during operation. Current methods for designing inductive power transfer (IPT) coils rely heavily on FEM (finite-element methods) simulations to evaluate each potential design. Dynamic IPT design requires multiple simulation runs as the EV coil passes over the roadway coil. Identifying optimal designs is difficult because of the many conflicting specifications and objective functions that need to be considered, such as maximizing the output power while minimizing stray magnetic fields and the volume of windings and magnetic cores. This work introduces a new design optimization method for dynamic IPT systems that utilize generative neural networks. Deep learning is applied to create a generator of near-optimal design alternatives from random noise. Two neural networks are employed in the approach. The first neural network is trained from multiple FEM results through random sampling of the design space and then replaces FEM calculations, allowing rapid simulation and evaluation of alternative designs. By using the neural network as a surrogate model rather than FEM to evaluate designs, differentiable programming approaches may be applied to train the second neural network to generate better designs. This generative network is trained by minimizing a loss function based on the optimization criteria listed earlier. Alternative loss function based on combining multi-objective optimization methods are explored, including applying the following mathematical operations over the objective functions: the sum of squares, a product of means, and sums of combinations of pair-wise products. Compared to previous work [1], which employed genetic algorithm approaches, the generative network quickly learned to produce designs that pass all objective functions using the product of means, however, the design solutions lacked diversity. Interestingly, when considering all pairwise product combinations only a few worked in quickly learning to produce satisfactory solutions. Those combinations that worked had a lower solution production rate than the product of means but exhibited a higher diversity of solutions

Analyzing Self-similar and Fractal Properties of the C. Elegans Neural Network

The brain is one of the most studied and highly complex systems in the biological world. While much research has concentrated on studying the brain directly, our focus is the structure of the brain itself: at its core an interconnected network of nodes (neurons). A better understanding of the structural connectivity of the brain should elucidate some of its functional properties. In this paper we analyze the connectome of the nematode Caenorhabditis elegans. Consisting of only 302 neurons, it is one of the better-understood neural networks. Using a Laplacian Matrix of the 279-neuron “giant component” of the network, we use an eigenvalue counting function to look for fractal-like self similarity. This matrix representation is also used to plot visualizations of the neural network in eigenfunction coordinates. Small-world properties of the system are examined, including average path length and clustering coefficient. We test for localization of eigenfunctions, using graph energy and spacial variance on these functions. To better understand results, all calculations are also performed on random networks, branching trees, and known fractals, as well as fractals which have been “rewired” to have small-world properties. We propose algorithms for generating Laplacian matrices of each of these graphs

Physics-guided machine learning approaches to predict the ideal stability properties of fusion plasmas

One of the biggest challenges to achieve the goal of producing fusion energy in tokamak devices is the necessity of avoiding disruptions of the plasma current due to instabilities. The disruption event characterization and forecasting (DECAF) framework has been developed in this purpose, integrating physics models of many causal events that can lead to a disruption. Two different machine learning approaches are proposed to improve the ideal magnetohydrodynamic (MHD) no-wall limit component of the kinetic stability model included in DECAF. First, a random forest regressor (RFR), was adopted to reproduce the DCON computed change in plasma potential energy without wall effects, , for a large database of equilibria from the national spherical torus experiment (NSTX). This tree-based method provides an analysis of the importance of each input feature, giving an insight into the underlying physics phenomena. Secondly, a fully-connected neural network has been trained on sets of calculations with the DCON code, to get an improved closed form equation of the no-wall limit as a function of the relevant plasma parameters indicated by the RFR. The neural network has been guided by physics theory of ideal MHD in its extension outside the domain of the NSTX experimental data. The estimated value of has been incorporated into the DECAF kinetic stability model and tested against a set of experimentally stable and unstable discharges. Moreover, the neural network results were used to simulate a real-time stability assessment using only quantities available in real-time. Finally, the portability of the model was investigated, showing encouraging results by testing the NSTX-trained algorithm on the mega ampere spherical tokamak (MAST)

Short-term forecasting and uncertainty analysis of wind turbine power based on long short-term memory network and Gaussian mixture model

Wind power plays a leading role in the development of renewable energy. However, the random nature of wind turbine power and its associated uncertainty create challenges in dispatching this power effectively in the power system, which can result in unnecessary curtailment of the wind turbine power. Improving the accuracy of wind turbine power forecasting is an effective measure for resolving such problems. This study uses a deep learning network to forecast the wind turbine power based on a long short-term memory network (LSTM) algorithm and uses the Gaussian mixture model (GMM) to analyze the error distribution characteristics of short-term wind turbine power forecasting. The LSTM algorithm is used to forecast the power and uncertainties for three wind turbines within a wind farm. According to numerical weather prediction (NWP) data and historical power data for three turbines, the forecasting accuracy of the turbine with the largest number of training samples is the best of the three. For one of the turbines, the LSTM, radial basis function (RBF), wavelet, deep belief network (DBN), back propagation neural networks (BPNN), and Elman neural network (ELMAN) have been used to forecast the wind turbine power. This study compares the results and demonstrates that LSTM can greatly improve the forecasting accuracy. Moreover, this study obtains different confidence intervals for the three units according to the GMM, mixture density neural network (MDN), and relevance vector machine (RVM) model results. The LSTM method is shown to have higher accuracy and faster convergence than the other methods. However, the GMM method has better performance and evaluation than other methods and thus has practical application value for wind turbine power dispatching