Advancements in Safe Deep Reinforcement Learning for Real-Time Strategy Games and Industry Applications

publishedVersio

Learning Augmented Optimization for Network Softwarization in 5G

The rapid uptake of mobile devices and applications are posing unprecedented traffic burdens on the existing networking infrastructures. In order to maximize both user experience and investment return, the networking and communications systems are evolving to the next gen- eration – 5G, which is expected to support more flexibility, agility, and intelligence towards provisioned services and infrastructure management. Fulfilling these tasks is challenging, as nowadays networks are increasingly heterogeneous, dynamic and expanded with large sizes. Network softwarization is one of the critical enabling technologies to implement these requirements in 5G. In addition to these problems investigated in preliminary researches about this technology, many new emerging application requirements and advanced opti- mization & learning technologies are introducing more challenges & opportunities for its fully application in practical production environment. This motivates this thesis to develop a new learning augmented optimization technology, which merges both the advanced opti- mization and learning techniques to meet the distinct characteristics of the new application environment. To be more specific, the abstracts of the key contents in this thesis are listed as follows: • We first develop a stochastic solution to augment the optimization of the Network Function Virtualization (NFV) services in dynamical networks. In contrast to the dominant NFV solutions applied for the deterministic networking environments, the inherent network dynamics and uncertainties from 5G infrastructure are impeding the rollout of NFV in many emerging networking applications. Therefore, Chapter 3 investigates the issues of network utility degradation when implementing NFV in dynamical networks, and proposes a robust NFV solution with full respect to the underlying stochastic features. By exploiting the hierarchical decision structures in this problem, a distributed computing framework with two-level decomposition is designed to facilitate a distributed implementation of the proposed model in large-scale networks. • Next, Chapter 4 aims to intertwin the traditional optimization and learning technologies. In order to reap the merits of both optimization and learning technologies but avoid their limitations, promissing integrative approaches are investigated to combine the traditional optimization theories with advanced learning methods. Subsequently, an online optimization process is designed to learn the system dynamics for the network slicing problem, another critical challenge for network softwarization. Specifically, we first present a two-stage slicing optimization model with time-averaged constraints and objective to safeguard the network slicing operations in time-varying networks. Directly solving an off-line solution to this problem is intractable since the future system realizations are unknown before decisions. To address this, we combine the historical learning and Lyapunov stability theories, and develop a learning augmented online optimization approach. This facilitates the system to learn a safe slicing solution from both historical records and real-time observations. We prove that the proposed solution is always feasible and nearly optimal, up to a constant additive factor. Finally, simulation experiments are also provided to demonstrate the considerable improvement of the proposals. • The success of traditional solutions to optimizing the stochastic systems often requires solving a base optimization program repeatedly until convergence. For each iteration, the base program exhibits the same model structure, but only differing in their input data. Such properties of the stochastic optimization systems encourage the work of Chapter 5, in which we apply the latest deep learning technologies to abstract the core structures of an optimization model and then use the learned deep learning model to directly generate the solutions to the equivalent optimization model. In this respect, an encoder-decoder based learning model is developed in Chapter 5 to improve the optimization of network slices. In order to facilitate the solving of the constrained combinatorial optimization program in a deep learning manner, we design a problem-specific decoding process by integrating program constraints and problem context information into the training process. The deep learning model, once trained, can be used to directly generate the solution to any specific problem instance. This avoids the extensive computation in traditional approaches, which re-solve the whole combinatorial optimization problem for every instance from the scratch. With the help of the REINFORCE gradient estimator, the obtained deep learning model in the experiments achieves significantly reduced computation time and optimality loss

Optimal Management of community Demand Response

More than one-third of the electricity produced globally is consumed by the residential sectors [1], with nearly 17% of CO2 emissions, are coming from residential buildings according to reports from 2018 [2] [3]. In order to cope with increase in electricity demand and consumption, while considering the environmental impacts, electricity providers are seeking to implement solutions to help them balance the supply with the electricity demand while mitigating emissions. Thus, increasing the number of conventional generation units and using unreliable renewable source of energy is not a viable investment. That’s why, in recent years research attention has shifted to demand side solutions [4]. This research investigates the optimal management for an urban residential community, that can help in reducing energy consumption and peak and CO2 emissions. This will help to put an agreement with the grid operator for an agreed load shape, for efficient demand response (DR) program implementation. This work uses a framework known as CityLearn [2]. It is based on a Machine Learning branch known as Reinforcement Learning (RL), and it is used to test a variety of intelligent agents for optimizing building load consumption and load shape. The RL agent is used for controlling hot water and chilled water storages, as well as the battery system. When compared to the regular building usage, the results demonstrate that utilizing an RL agent for storage system control can be helpful, as the electricity consumption is greatly reduced when it’s compared to the normal building consumption

Trajectory Planning and Subject-Specific Control of a Stroke Rehabilitation Robot using Deep Reinforcement Learning

There are approximately 13 million annual new stroke cases worldwide. Research has shown that robotics can provide practical and efficient solutions for expediting post-stroke patient recovery. Assistive robots provide automatic limb training, which saves a great deal of time and energy. In addition, they facilitate the use of data acquisition devices. The data is beneficial in terms of quantitative evaluation of the patient progress. This research focused on the trajectory planning and subject-specific control of an upper-extremity post-stroke rehabilitation robot. To find the optimal rehabilitation practice, the manipulation trajectory was designed by an optimization-based planner. A linear quadratic regulator (LQR) controller was then applied to stabilize the trajectory. The integrated planner-controller framework was tested in simulation. To validate the simulation results, hardware implementation was conducted, which provided good agreement with simulation. One of the challenges of rehabilitation robotics is the choice of the low-level controller. To find the best candidate for our specific setup, five controllers were evaluated in simulation for circular trajectory tracking. In particular, we compared the performance of LQR, sliding mode control (SMC), and nonlinear model predictive control (NMPC) to conventional proportional integral derivative (PID) and computed-torque PID controllers. The real-time assessment of the mentioned controllers was done by implementing them on the physical hardware for point stabilization and circular trajectory tracking scenarios. Our comparative study confirmed the need for advanced low-level controllers for better performance. Due to complex online optimization of the NMPC and the incorporated delay in the method of implementation, performance degradation was observed with NMPC compared to other advanced controllers. The evaluation showed that SMC and LQR were the two best candidates for the robot. To remove the need for extensive manual controller tuning, a deep reinforcement learning (DRL) tuner framework was designed in MATLAB to provide the optimal weights for the controllers; it permitted the online tuning of the weights, which enabled the subject-specific controller weight adjustment. This tuner was tested in simulation by adding a random noise to the input at each iteration, to simulate the subject. Compared to fixed manually tuned weights, the DRL-tuned controller presented lower position-error. In addition, an easy to implement high-level force controller algorithm was designed by incorporating the subject force data. The resulting hybrid position/force controller was tested with a healthy subject in the loop. The controller was able to provide assist as needed when the subject increased the position-error. Future research might consider model reduction methods for expediting the NMPC optimization, application of the DRL on other controllers and for optimization parameter adjustment, testing other high-level controllers like admittance control, and testing the final controllers with post-stroke patients

Computational intelligence approaches to robotics, automation, and control [Volume guest editors]

No abstract available

Machine Learning

Machine Learning can be defined in various ways related to a scientific domain concerned with the design and development of theoretical and implementation tools that allow building systems with some Human Like intelligent behavior. Machine learning addresses more specifically the ability to improve automatically through experience

Modelling, Monitoring, Control and Optimization for Complex Industrial Processes

This reprint includes 22 research papers and an editorial, collected from the Special Issue "Modelling, Monitoring, Control and Optimization for Complex Industrial Processes", highlighting recent research advances and emerging research directions in complex industrial processes. This reprint aims to promote the research field and benefit the readers from both academic communities and industrial sectors