Incremental construction of LSTM recurrent neural network

Long Short--Term Memory (LSTM) is a recurrent neural network that uses structures called memory blocks to allow the net remember significant events distant in the past input sequence in order to solve long time lag tasks, where other RNN approaches fail. Throughout this work we have performed experiments using LSTM networks extended with growing abilities, which we call GLSTM. Four methods of training growing LSTM has been compared. These methods include cascade and fully connected hidden layers as well as two different levels of freezing previous weights in the cascade case. GLSTM has been applied to a forecasting problem in a biomedical domain, where the input/output behavior of five controllers of the Central Nervous System control has to be modelled. We have compared growing LSTM results against other neural networks approaches, and our work applying conventional LSTM to the task at hand.Postprint (published version

GAdaboost: Accelerating adaboost feature selection with genetic algorithms

Throughout recent years Machine Learning has acquired attention, due to the abundant data. Thus, devising techniques to reduce the dimensionality of data has been on going. Object detection is one of the Machine Learning techniques which suffer from this draw back. As an example, one of the most famous object detection frameworks is the Viola-Jones Rapid Object Detector, which suffers from a lengthy training process due to the vast search space, which can reach more than 160,000 features for a 24X24 image. The Viola-Jones Rapid Object Detector also uses Adaboost, which is a brute force method, and is required to pass by the set of all possible features in order to train the classifiers. Consequently, ways for reducing the whole feature set into a smaller representative one, eliminating those features that have non relevant information, were devised. The most commonly used technique for this is Feature Selection with its three categories: Filters, Wrappers and Embedded. Feature Selection has proven its success in providing fast and accurate classifiers. Wrapper methods harvest the power of evolutionary computing, most commonly Genetic Algorithms, in finding the set of representative features. This is mostly due to the Advantage of Genetic Algorithms and their power in finding adequate solutions more efficiently. In this thesis we propose GAdaboost: A Genetic Algorithm to accelerate the training procedure of the Viola-Jones Rapid Object Detector through Feature Selection. Specifically, we propose to limit the Adaboost search within a sub-set of the huge feature space, while evolving this subset following a Genetic Algorithm. Experiments demonstrate that our proposed GAdaboost is up to 3.7 times faster than Adaboost. We also demonstrate that the price of this speedup is a mere decrease (3%, 4%) in detection accuracy when tested on FDDB benchmark face detection set, and Caltech Web Faces respectivel

Cuckoo Search Inspired Hybridization of the Nelder-Mead Simplex Algorithm Applied to Optimization of Photovoltaic Cells

A new hybridization of the Cuckoo Search (CS) is developed and applied to optimize multi-cell solar systems; namely multi-junction and split spectrum cells. The new approach consists of combining the CS with the Nelder-Mead method. More precisely, instead of using single solutions as nests for the CS, we use the concept of a simplex which is used in the Nelder-Mead algorithm. This makes it possible to use the flip operation introduces in the Nelder-Mead algorithm instead of the Levy flight which is a standard part of the CS. In this way, the hybridized algorithm becomes more robust and less sensitive to parameter tuning which exists in CS. The goal of our work was to optimize the performance of multi-cell solar systems. Although the underlying problem consists of the minimization of a function of a relatively small number of parameters, the difficulty comes from the fact that the evaluation of the function is complex and only a small number of evaluations is possible. In our test, we show that the new method has a better performance when compared to similar but more compex hybridizations of Nelder-Mead algorithm using genetic algorithms or particle swarm optimization on standard benchmark functions. Finally, we show that the new method outperforms some standard meta-heuristics for the problem of interest

Turn-Key Stabilization and Digital Control of Scalable, N GTI Resonator Based Coherent Pulse Stacking Systems

Coherent Pulse Stacking Amplification (CPSA) is a new time-domain coherent addition technique that overcomes the limitations on pulse energies achievable from optical amplifiers. It uses reflecting resonators to transform a sequence of phase- and amplitude-modulated optical pulses into a single output pulse enabling high pulse energy for fiber lasers. This thesis focuses on utilizing efficient algorithms for stabilization and optimization aspects of CPSA and developing a robust, scalable, and distributed digital control system with firmware and software integration for algorithms, to support the CPS (Coherent Pulse Stacking) application. We have presented the theoretical foundation of the stochastic parallel gradient descent (SPGD) for phase stabilization, discussed its performance criteria, its convergence, and its stability. We have presented our software and hardware development for time-domain coherent combing stabilization (specifically, an FPGA (Field Programmable Gate Array)-based Control system with software/firmware development to support stabilization and optimization algorithms). Analytical formulations of output stacked pulse profile as a function of input pulse train amplitudes and phase and stacker cavity parameters have been derived so as to build up a foundation for a GTI (Gires-Tournois-Interferometer) Cavity-based noise measurement technique. Time-domain and frequency domain characterization techniques have been presented to analyze phase and amplitude noise in the stacking system. Stacking sensitivity to errors in different control parameters (stacker cavity phase, pulse amplitude, and phases) for different stacker configurations have been analyzed. Noise measurement results using GTI cavities with different round-trip time has have been presented and we have shown how effectively the stacking phase noise in the system can be reduced by improving the noise performance of the mode-locked oscillator. Simulation and Experimental results for stabilizing different stacker configurations have been presented. Finally an algorithmic control system along with software/hardware development for optimizing amplitudes and phases of the input burst has been implemented to increase stacking fidelity. A complete detailed description, and simulation of the Genetic Algorithm as an alternative algorithm for optimizing the stacked pulse fidelity has been presented. Comparison between SPGD and Genetic Algorithm results has been done to evaluate their performance. To summarize, this thesis provides theoretical, experimental, and implementation aspects of controlling CPSA system by introducing efficient control algorithms and developing a turn-key digital control system which is scalable to large number of stacker cavities.PHDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/147664/1/msheikhs_1.pd

Adaptive non linear system identification and channel equalization usinf functional link artificial neural network

In system theory, characterization and identification are fundamental problems. When the plant behavior is completely unknown, it may be characterized using certain model and then, its identification may be carried out with some artificial neural networks(ANN) like multilayer perceptron(MLP) or functional link artificial neural network(FLANN) using some learning rules such as back propagation (BP) algorithm. They offer flexibility, adaptability and versatility, so that a variety of approaches may be used to meet a specific goal, depending upon the circumstances and the requirements of the design specifications. The primary aim of the present thesis is to provide a framework for the systematic design of adaptation laws for nonlinear system identification and channel equalization. While constructing an artificial neural network the designer is often faced with the problem of choosing a network of the right size for the task. The advantages of using a smaller neural network are cheaper cost of computation and better generalization ability. However, a network which is too small may never solve the problem, while a larger network may even have the advantage of a faster learning rate. Thus it makes sense to start with a large network and then reduce its size. For this reason a Genetic Algorithm (GA) based pruning strategy is reported. GA is based upon the process of natural selection and does not require error gradient statistics. As a consequence, a GA is able to find a global error minimum. Transmission bandwidth is one of the most precious resources in digital communication systems. Communication channels are usually modeled as band-limited linear finite impulse response (FIR) filters with low pass frequency response. When the amplitude and the envelope delay response are not constant within the bandwidth of the filter, the channel distorts the transmitted signal causing intersymbol interference (ISI). The addition of noise during propagation also degrades the quality of the received signal. All the signal processing methods used at the receiver's end to compensate the introduced channel distortion and recover the transmitted symbols are referred as channel equalization techniques.When the nonlinearity associated with the system or the channel is more the number of branches in FLANN increases even some cases give poor performance. To decrease the number of branches and increase the performance a two stage FLANN called cascaded FLANN (CFLANN) is proposed.This thesis presents a comprehensive study covering artificial neural network (ANN) implementation for nonlinear system identification and channel equalization. Three ANN structures, MLP, FLANN, CFLANN and their conventional gradient-descent training methods are extensively studied. Simulation results demonstrate that FLANN and CFLANN methods are directly applicable for a large class of nonlinear control systems and communication problems

Indoor Wireless RF Energy Transfer for Powering Wireless Sensors

For powering wireless sensors in buildings, rechargeable batteries may be used. These batteries will be recharged remotely by dedicated RF sources. Far-field RF energy transport is known to suffer from path loss and therefore the RF power available on the rectifying antenna or rectenna will be very low. As a consequence, the RF-to-DC conversion efficiency of the rectenna will also be very low. By optimizing not only the subsystems of a rectenna but also taking the propagation channel into account and using the channel information for adapting the transmit antenna radiation pattern, the RF energy transport efficiency will be improved. The rectenna optimization, channel modeling and design of a transmit antenna are discussed