Advances in Reinforcement Learning

Reinforcement Learning (RL) is a very dynamic area in terms of theory and application. This book brings together many different aspects of the current research on several fields associated to RL which has been growing rapidly, producing a wide variety of learning algorithms for different applications. Based on 24 Chapters, it covers a very broad variety of topics in RL and their application in autonomous systems. A set of chapters in this book provide a general overview of RL while other chapters focus mostly on the applications of RL paradigms: Game Theory, Multi-Agent Theory, Robotic, Networking Technologies, Vehicular Navigation, Medicine and Industrial Logistic

Load frequency controllers considering renewable energy integration in power system

Abstract: Load frequency control or automatic generation control is one of the main operations that take place daily in a modern power system. The objectives of load frequency control are to maintain power balance between interconnected areas and to control the power flow in the tie-lines. Electric power cannot be stored in large quantity that is why its production must be equal to the consumption in each time. This equation constitutes the key for a good management of any power system and introduces the need of more controllers when taking into account the integration of renewable energy sources into the traditional power system. There are many controllers presented in the literature and this work reviews the traditional load frequency controllers and those, which combined the traditional controller and artificial intelligence algorithms for controlling the load frequency

Design and real-time implementation of data-driven adaptive wide-area damping controller for back-to-back VSC-HVDC

This paper proposes a data-driven adaptive wide-area damping controller (D-WADC) for back-to-back VSC-HVDC to suppress the low frequency oscillation in a large-scale interconnected power system. The proposed D-WADC adopts a dual-loop control structure to make full use of the active and reactive power control of VSC-HVDC to improve the damping of the power system. A data-driven algorithm named the goal representation heuristic dynamic programming is employed to design the proposed D-WADC, which means the design procedure only requires the input and output data rather than the mathematic model of the concerned power system. Thus, the D-WADC can adapt to the change of operating condition through online weight modification. Besides, the adaptive delay compensator (ADC) is added to effectively compensate the stochastic delay involved in the wide-area feedback signal. Case studies are conducted based on the simplified model of a practical power system and the 16-machine system with a back-to-back VSC-HVDC. Both the simulation and hardware-in-loop experiment results verify that the proposed D-WADC can effectively suppress the low-frequency oscillation under a wide range of operating conditions, disturbances, and stochastic communication delays

Chaotic exploration and learning of locomotion behaviours

We present a general and fully dynamic neural system, which exploits intrinsic chaotic dynamics, for the real-time goal-directed exploration and learning of the possible locomotion patterns of an articulated robot of an arbitrary morphology in an unknown environment. The controller is modeled as a network of neural oscillators that are initially coupled only through physical embodiment, and goal-directed exploration of coordinated motor patterns is achieved by chaotic search using adaptive bifurcation. The phase space of the indirectly coupled neural-body-environment system contains multiple transient or permanent self-organized dynamics, each of which is a candidate for a locomotion behavior. The adaptive bifurcation enables the system orbit to wander through various phase-coordinated states, using its intrinsic chaotic dynamics as a driving force, and stabilizes on to one of the states matching the given goal criteria. In order to improve the sustainability of useful transient patterns, sensory homeostasis has been introduced, which results in an increased diversity of motor outputs, thus achieving multiscale exploration. A rhythmic pattern discovered by this process is memorized and sustained by changing the wiring between initially disconnected oscillators using an adaptive synchronization method. Our results show that the novel neurorobotic system is able to create and learn multiple locomotion behaviors for a wide range of body configurations and physical environments and can readapt in realtime after sustaining damage

DiffTune: Auto-Tuning through Auto-Differentiation

The performance of robots in high-level tasks depends on the quality of their lower-level controller, which requires fine-tuning. However, the intrinsically nonlinear dynamics and controllers make tuning a challenging task when it is done by hand. In this paper, we present DiffTune, a novel, gradient-based automatic tuning framework. We formulate the controller tuning as a parameter optimization problem. Our method unrolls the dynamical system and controller as a computational graph and updates the controller parameters through gradient-based optimization. The gradient is obtained using sensitivity propagation, which is the only method for gradient computation when tuning for a physical system instead of its simulated counterpart. Furthermore, we use $\mathcal{L}_1$ adaptive control to compensate for the uncertainties (that unavoidably exist in a physical system) such that the gradient is not biased by the unmodelled uncertainties. We validate the DiffTune on a Dubin's car and a quadrotor in challenging simulation environments. In comparison with state-of-the-art auto-tuning methods, DiffTune achieves the best performance in a more efficient manner owing to its effective usage of the first-order information of the system. Experiments on tuning a nonlinear controller for quadrotor show promising results, where DiffTune achieves 3.5x tracking error reduction on an aggressive trajectory in only 10 trials over a 12-dimensional controller parameter space.Comment: Minkyung Kim and Lin Song contributed equally to this wor

Bio-inspired Hardware Architectures for Memory, Image Processing, and Control Applications

Emerging technologies are expected to partially replace and enhance CMOS systems as the end of transistor scaling approaches. A particular type of emerging technology of interest is the variable resistance devices due to their scalability, non-volatile nature, and CMOS process compatibility. The goal of this dissertation is to present circuit and system level applications of CMOS and variable resistance devices with bio-inspired computation paradigms as the main focus. The summary of the results offered per chapter is as follows: In the first chapter of this thesis, an introduction to the work presented in the rest of this thesis and the model for the variable resistance device is provided. In the second chapter of this thesis, a crossbar memory architecture that utilizes a reduced constraint read-monitored-write scheme is presented. Variable resistance based crossbar memories are prime candidates to succeed the Flash as the mainstream nonvolatile memory due to their density, scalability, and write endurance. The proposed scheme supports multi-bit storage per cell and utilizes reduced hardware, aiming to decrease the feedback complexity and latency while still operating with CMOS compatible voltages. Additionally, a read technique that can successfully distinguish resistive states under the existence of resistance drift due to read/write disturbances in the array is presented. Derivations of analytical relations are provided to set forth a design methodology in selecting peripheral device parameters. In the third chapter of this thesis, an analog programmable resistive grid-based architecture mimicking the cellular connections of a biological retina in the most basic level, capable of performing various real time image processing tasks such as edge and line detections, is presented. Resistive grid-based analog structures have been shown to have advantages of compact area, noise immunity, and lower power consumption compared to their digital counterparts. However, these are static structures that can only perform one type of image processing task. The proposed unit cell structure employs 3-D confined resonant tunneling diodes called quantum dots for signal amplification and latching, and these dots are interconnected between neighboring cells through non-volatile continuously variable resistive elements. A method to program connections is introduced and verified through circuit simulations. Various diffusion characteristics, edge detection, and line detection tasks have been demonstrated through simulations using a 2-D array of the proposed cell structure, and analytical models have been provided. In the fourth chapter of this thesis, a bio-inspired hardware designed to solve the optimal control problem for general systems is presented. Adaptive Dynamic Programming algorithms provide means to approximate optimal control actions for linear and non-linear systems. Action-Critic Networks based approach is an efficient way to approximately evaluate the cost function and the optimal control actions. However, due to its computation intensiveness, this approach is usually implemented in high level programming languages run using general purpose processors. The presented hardware design is aimed at reducing the computation time and the hardware overhead by using the Heuristic Dynamic Programming algorithm which is a form of Adaptive Dynamic Programming. The proposed hardware operating at mere speed of 10 MHz yields 237 times faster learning rate in comparison to conventional software implementations running on fast processors such as the 1.2 GHz Intel Xeon processor.PHDElectrical EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/136972/1/yalciny_1.pd

Advances in Condition Monitoring, Optimization and Control for Complex Industrial Processes

The book documents 25 papers collected from the Special Issue “Advances in Condition Monitoring, Optimization and Control for Complex Industrial Processes”, highlighting recent research trends in complex industrial processes. The book aims to stimulate the research field and be of benefit to readers from both academic institutes and industrial sectors