An FPGA-Based On-Device Reinforcement Learning Approach using Online Sequential Learning

DQN (Deep Q-Network) is a method to perform Q-learning for reinforcement learning using deep neural networks. DQNs require a large buffer and batch processing for an experience replay and rely on a backpropagation based iterative optimization, making them difficult to be implemented on resource-limited edge devices. In this paper, we propose a lightweight on-device reinforcement learning approach for low-cost FPGA devices. It exploits a recently proposed neural-network based on-device learning approach that does not rely on the backpropagation method but uses OS-ELM (Online Sequential Extreme Learning Machine) based training algorithm. In addition, we propose a combination of L2 regularization and spectral normalization for the on-device reinforcement learning so that output values of the neural network can be fit into a certain range and the reinforcement learning becomes stable. The proposed reinforcement learning approach is designed for PYNQ-Z1 board as a low-cost FPGA platform. The evaluation results using OpenAI Gym demonstrate that the proposed algorithm and its FPGA implementation complete a CartPole-v0 task 29.77x and 89.40x faster than a conventional DQN-based approach when the number of hidden-layer nodes is 64

A new Sigma-Pi-Sigma neural network based on L1 L_1 and L2 L_2 regularization and applications

As one type of the important higher-order neural networks developed in the last decade, the Sigma-Pi-Sigma neural network has more powerful nonlinear mapping capabilities compared with other popular neural networks. This paper is concerned with a new Sigma-Pi-Sigma neural network based on a $L_1$ and $L_2$ regularization batch gradient method, and the numerical experiments for classification and regression problems prove that the proposed algorithm is effective and has better properties comparing with other classical penalization methods. The proposed model combines the sparse solution tendency of $L_1$ norm and the high benefits in efficiency of the $L_2$ norm, which can regulate the complexity of a network and prevent overfitting. Also, the numerical oscillation, induced by the non-differentiability of $L_1$ plus $L_2$ regularization at the origin, can be eliminated by a smoothing technique to approximate the objective function

A simulation data-driven design approach for rapid product optimization

Traditional design optimization is an iterative process of design, simulation, and redesign, which requires extensive calculations and analysis. The designer needs to adjust and evaluate the design parameters manually and continually based on the simulation results until a satisfactory design is obtained. However, the expensive computational costs and large resource consumption of complex products hinder the wide application of simulation in industry. It is not an easy task to search the optimal design solution intelligently and efficiently. Therefore, a simulation data-driven design approach which combines dynamic simulation data mining and design optimization is proposed to achieve this purpose in this study. The dynamic simulation data mining algorithm—on-line sequential extreme learning machine with adaptive weights (WadaptiveOS-ELM)—is adopted to train the dynamic prediction model to effectively evaluate the merits of new design solutions in the optimization process. Meanwhile, the prediction model is updated incrementally by combining new “good” data set to reduce the modeling cost and improve the prediction accuracy. Furthermore, the improved heuristic optimization algorithm—adaptive and weighted center particle swarm optimization (AWCPSO)—is introduced to guide the design change direction intelligently to improve the search efficiency. In this way, the optimal design solution can be searched automatically with less actual simulation iterations and higher optimization efficiency, and thus supporting the rapid product optimization effectively. The experimental results demonstrate the feasibility and effectiveness of the proposed approach

Adaptive Control Using Fully Online Sequential-Extreme Learning Machine and a Case Study on Engine Air-Fuel Ratio Regulation

Most adaptive neural control schemes are based on stochastic gradient-descent backpropagation (SGBP), which suffers from local minima problem. Although the recently proposed regularized online sequential-extreme learning machine (ReOS-ELM) can overcome this issue, it requires a batch of representative initial training data to construct a base model before online learning. The initial data is usually difficult to collect in adaptive control applications. Therefore, this paper proposes an improved version of ReOS-ELM, entitled fully online sequential-extreme learning machine (FOS-ELM). While retaining the advantages of ReOS-ELM, FOS-ELM discards the initial training phase, and hence becomes suitable for adaptive control applications. To demonstrate its effectiveness, FOS-ELM was applied to the adaptive control of engine air-fuel ratio based on a simulated engine model. Besides, controller parameters were also analyzed, in which it is found that large hidden node number with small regularization parameter leads to the best performance. A comparison among FOS-ELM and SGBP was also conducted. The result indicates that FOS-ELM achieves better tracking and convergence performance than SGBP, since FOS-ELM tends to learn the unknown engine model globally whereas SGBP tends to “forget” what it has learnt. This implies that FOS-ELM is more preferable for adaptive control applications

Online Regularized and Kernelized Extreme Learning Machines with Forgetting Mechanism

To apply the single hidden-layer feedforward neural networks (SLFN) to identify time-varying system, online regularized extreme learning machine (ELM) with forgetting mechanism (FORELM) and online kernelized ELM with forgetting mechanism (FOKELM) are presented in this paper. The FORELM updates the output weights of SLFN recursively by using Sherman-Morrison formula, and it combines advantages of online sequential ELM with forgetting mechanism (FOS-ELM) and regularized online sequential ELM (ReOS-ELM); that is, it can capture the latest properties of identified system by studying a certain number of the newest samples and also can avoid issue of ill-conditioned matrix inversion by regularization. The FOKELM tackles the problem of matrix expansion of kernel based incremental ELM (KB-IELM) by deleting the oldest sample according to the block matrix inverse formula when samples occur continually. The experimental results show that the proposed FORELM and FOKELM have better stability than FOS-ELM and have higher accuracy than ReOS-ELM in nonstationary environments; moreover, FORELM and FOKELM have time efficiencies superiority over dynamic regression extreme learning machine (DR-ELM) under certain conditions

CPS Data Streams Analytics based on Machine Learning for Cloud and Fog Computing: A Survey

Cloud and Fog computing has emerged as a promising paradigm for the Internet of things (IoT) and cyber-physical systems (CPS). One characteristic of CPS is the reciprocal feedback loops between physical processes and cyber elements (computation, software and networking), which implies that data stream analytics is one of the core components of CPS. The reasons for this are: (i) it extracts the insights and the knowledge from the data streams generated by various sensors and other monitoring components embedded in the physical systems; (ii) it supports informed decision making; (iii) it enables feedback from the physical processes to the cyber counterparts; (iv) it eventually facilitates the integration of cyber and physical systems. There have been many successful applications of data streams analytics, powered by machine learning techniques, to CPS systems. Thus, it is necessary to have a survey on the particularities of the application of machine learning techniques to the CPS domain. In particular, we explore how machine learning methods should be deployed and integrated in cloud and fog architectures for better fulfilment of the requirements, e.g. mission criticality and time criticality, arising in CPS domains. To the best of our knowledge, this paper is the first to systematically study machine learning techniques for CPS data stream analytics from various perspectives, especially from a perspective that leads to the discussion and guidance of how the CPS machine learning methods should be deployed in a cloud and fog architecture

SYSTEM-ON-A-CHIP (SOC)-BASED HARDWARE ACCELERATION FOR HUMAN ACTION RECOGNITION WITH CORE COMPONENTS

Today, the implementation of machine vision algorithms on embedded platforms or in portable systems is growing rapidly due to the demand for machine vision in daily human life. Among the applications of machine vision, human action and activity recognition has become an active research area, and market demand for providing integrated smart security systems is growing rapidly. Among the available approaches, embedded vision is in the top tier; however, current embedded platforms may not be able to fully exploit the potential performance of machine vision algorithms, especially in terms of low power consumption. Complex algorithms can impose immense computation and communication demands, especially action recognition algorithms, which require various stages of preprocessing, processing and machine learning blocks that need to operate concurrently. The market demands embedded platforms that operate with a power consumption of only a few watts. Attempts have been mad to improve the performance of traditional embedded approaches by adding more powerful processors; this solution may solve the computation problem but increases the power consumption. System-on-a-chip eld-programmable gate arrays (SoC-FPGAs) have emerged as a major architecture approach for improving power eciency while increasing computational performance. In a SoC-FPGA, an embedded processor and an FPGA serving as an accelerator are fabricated in the same die to simultaneously improve power consumption and performance. Still, current SoC-FPGA-based vision implementations either shy away from supporting complex and adaptive vision algorithms or operate at very limited resolutions due to the immense communication and computation demands. The aim of this research is to develop a SoC-based hardware acceleration workflow for the realization of advanced vision algorithms. Hardware acceleration can improve performance for highly complex mathematical calculations or repeated functions. The performance of a SoC system can thus be improved by using hardware acceleration method to accelerate the element that incurs the highest performance overhead. The outcome of this research could be used for the implementation of various vision algorithms, such as face recognition, object detection or object tracking, on embedded platforms. The contributions of SoC-based hardware acceleration for hardware-software codesign platforms include the following: (1) development of frameworks for complex human action recognition in both 2D and 3D; (2) realization of a framework with four main implemented IPs, namely, foreground and background subtraction (foreground probability), human detection, 2D/3D point-of-interest detection and feature extraction, and OS-ELM as a machine learning algorithm for action identication; (3) use of an FPGA-based hardware acceleration method to resolve system bottlenecks and improve system performance; and (4) measurement and analysis of system specications, such as the acceleration factor, power consumption, and resource utilization. Experimental results show that the proposed SoC-based hardware acceleration approach provides better performance in terms of the acceleration factor, resource utilization and power consumption among all recent works. In addition, a comparison of the accuracy of the framework that runs on the proposed embedded platform (SoCFPGA) with the accuracy of other PC-based frameworks shows that the proposed approach outperforms most other approaches

Electricity Price Time Series Forecasting in Deregulated Markets Using Recurrent Neural Network Based Approaches

Ph.DDOCTOR OF PHILOSOPH

Machine Learning for Identification and Optimal Control of Advanced Automotive Engines.

The complexity of automotive engines continues to increase to meet increasing performance requirements such as high fuel economy and low emissions. The increased sensing capabilities associated with such systems generate a large volume of informative data. With advancements in computing technologies, predictive models of complex dynamic systems useful for diagnostics and controls can be developed using data based learning. Such models have a short development time and can serve as alternatives to traditional physics based modeling. In this thesis, the modeling and control problem of an advanced automotive engine, the homogeneous charge compression ignition (HCCI) engine, is addressed using data based learning techniques. Several frameworks including design of experiments for data generation, identification of HCCI combustion variables, modeling the HCCI operating envelope and model predictive control have been developed and analyzed. In addition, stable online learning algorithms for a general class of nonlinear systems have been developed using extreme learning machine (ELM) model structure.PHDMechanical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttp://deepblue.lib.umich.edu/bitstream/2027.42/102392/1/vijai_1.pd