Computer Architectures to Close the Loop in Real-time Optimization

© 2015 IEEE.Many modern control, automation, signal processing and machine learning applications rely on solving a sequence of optimization problems, which are updated with measurements of a real system that evolves in time. The solutions of each of these optimization problems are then used to make decisions, which may be followed by changing some parameters of the physical system, thereby resulting in a feedback loop between the computing and the physical system. Real-time optimization is not the same as fast optimization, due to the fact that the computation is affected by an uncertain system that evolves in time. The suitability of a design should therefore not be judged from the optimality of a single optimization problem, but based on the evolution of the entire cyber-physical system. The algorithms and hardware used for solving a single optimization problem in the office might therefore be far from ideal when solving a sequence of real-time optimization problems. Instead of there being a single, optimal design, one has to trade-off a number of objectives, including performance, robustness, energy usage, size and cost. We therefore provide here a tutorial introduction to some of the questions and implementation issues that arise in real-time optimization applications. We will concentrate on some of the decisions that have to be made when designing the computing architecture and algorithm and argue that the choice of one informs the other

Deep Reinforcement Learning for Real-Time Optimization in NB-IoT Networks

NarrowBand-Internet of Things (NB-IoT) is an emerging cellular-based technology that offers a range of flexible configurations for massive IoT radio access from groups of devices with heterogeneous requirements. A configuration specifies the amount of radio resource allocated to each group of devices for random access and for data transmission. Assuming no knowledge of the traffic statistics, there exists an important challenge in "how to determine the configuration that maximizes the long-term average number of served IoT devices at each Transmission Time Interval (TTI) in an online fashion". Given the complexity of searching for optimal configuration, we first develop real-time configuration selection based on the tabular Q-learning (tabular-Q), the Linear Approximation based Q-learning (LA-Q), and the Deep Neural Network based Q-learning (DQN) in the single-parameter single-group scenario. Our results show that the proposed reinforcement learning based approaches considerably outperform the conventional heuristic approaches based on load estimation (LE-URC) in terms of the number of served IoT devices. This result also indicates that LA-Q and DQN can be good alternatives for tabular-Q to achieve almost the same performance with much less training time. We further advance LA-Q and DQN via Actions Aggregation (AA-LA-Q and AA-DQN) and via Cooperative Multi-Agent learning (CMA-DQN) for the multi-parameter multi-group scenario, thereby solve the problem that Q-learning agents do not converge in high-dimensional configurations. In this scenario, the superiority of the proposed Q-learning approaches over the conventional LE-URC approach significantly improves with the increase of configuration dimensions, and the CMA-DQN approach outperforms the other approaches in both throughput and training efficiency

Real time optimization of chemical processes

Due to current changes in the global market with increasing competition, strict bounds on product specifications, pricing pressures, and environmental issues, the chemical process industry has a high demand for methods and tools that enhance profitability by reducing the operating costs using limited resources. Real time optimization (RTO) strategies combine process control and economics, and have gone through much advancement during the last few decades. A typical real time optimization application is model based and requires the solution of at least three (usually) nonlinear programming problems, such as combined gross error detection and data reconciliation, parameter estimation and economic optimization. A successful implementation of RTO requires fast and accurate solution of these stated nonlinear programming problems.Current real time optimization strategies wait for steady state after a disturbance enters the process. If, during this wait, another disturbance enters into the system, it will increase the transition time significantly. An alternative, real time evolution (RTE), calculates the new set-points using only disturbance information and the new set-points are implemented in small step changes to a supervisory control system such as model predictive control (MPC) or can be implemented directly to the regulatory control layer. RTE ignores the important part of data screening therefore there is no surety that the calculated set-points represents current plant conditions. The main contribution of this thesis is to investigate the possibility of implementing new set-points without waiting for steady state. Two case studies, the Williams-Otto reactor and an integrated plant (the Williams-Otto reactor extended to include flash drum and large recycle stream), were used for analysis. The application of RTE, RTO and MPC were discussed and compared for the case studies to evaluate the performance in terms of the theoretical profit achieved.A new strategy, dynamic-RTO (D-RTO), based on modified dynamic data reconciliation (DDR) strategy and translated steady state model, was also developed for systems with significant bias and process noise. In the D-RTO strategy, the residual terms of the steady state model were calculated from the reconciled values. These residual terms were translated subsequently into the steady state model. Due to the translation there is no need for calculating set-point changes in small steps. The formulation of the DDR strategy is based on control vector parameterization techniques. D-RTO was compared with RTE and RTO for the two case studies. The results obtained show that RTE can lead to an unstable control if used without taking into account process and controller dynamics. For measurements having bias, the DDR strategy can be used with the assumption that the variables with bias are unmeasured and are calculated implicitly. The D-RTO strategy is able to deal with constant and changing bias, and is able to decrease profit losses during transitions. D-RTO is a good alternative to steady state RTO, for processes with frequent disturbances, where RTO implementation due to its steady state nature may not be justifiable

A Real-Time Optimization for 2R Manipulators

This work proposes a real-time algorithm to generate a trajectory for a 2 link planar robotic manipulator. The objective is to minimize the space/time ripple and the energy requirements or the time duration in the robot trajectories. The proposed method uses an off line genetic algorithm to calculate every possible trajectory between all cells of the workspace grid. The resultant trajectories are saved in several trees. Then any trajectory requested is constructed in real-time, from these trees. The article presents the results for several experiments

Real-Time Optimization for Dynamic Ride-Sharing

Throughout the last decade, the advent of novel mobility services such as ride-hailing, car-sharing, and ride-sharing has shaped urban mobility. While these types of services offer flexible on-demand transportation for customers, they may also increase the load on the, already strained, road infrastructure and exacerbate traffic congestion problems. One potential way to remedy this problem is the increased usage of dynamic ride-sharing services. In this type of service, multiple customer trips are combined into share a vehicle simultaneously. This leads to more efficient vehicle utilization, reduced prices for customers, and less traffic congestion at the cost of slight delays compared to direct transportation in ride-hailing services. In this thesis, we consider the planning and operation of such dynamic ride-sharing services. We present a wider look at the planning context of dynamic ride-sharing and discuss planning problems on the strategical, tactical, and operational level. Subsequently, our focus is on two operational planning problems: dynamic vehicle routing, and idle vehicle repositioning. Regarding vehicle routing, we introduce the vehicle routing problem for dynamic ridesharing and present a solution procedure. Our algorithmic approach consists of two phases: a fast insertion heuristic, and a local search improvement phase. The former handles incoming trip requests and quickly assigns them to suitable vehicles while the latter is responsible for continuously improving the current routing plan. This way, we enable fast response times for customers while simultaneously effectively utilizing available computational resources. Concerning the idle vehicle repositioning problem, we propose a mathematical model that takes repositioning decisions and adequately reflects available vehicle resources as well as a forecast of the upcoming trip request demand. This model is embedded into a real-time planning algorithm that regularly re-optimizes the movement of idle vehicles. Through an adaptive parameter calculation process, our algorithm dynamically adapts to changes in the current system state. To evaluate our algorithms, we present a modular simulation-based evaluation framework. We envision that this framework may also be used by other researchers and developers. In this thesis, we perform computational evaluations on a variety of scenarios based on real-world data from Chengdu, New York City, and Hamburg. The computational results show that we are able to produce high-quality solutions in real-time, enabling the usage in high-demand settings. In addition, our algorithms perform robustly in a variety of settings and are quickly adapted to new application settings, such as the deployment in a new city

Sufficient Conditions for Feasibility and Optimality of Real-Time Optimization Schemes - II. Implementation Issues

The idea of iterative process optimization based on collected output measurements, or "real-time optimization" (RTO), has gained much prominence in recent decades, with many RTO algorithms being proposed, researched, and developed. While the essential goal of these schemes is to drive the process to its true optimal conditions without violating any safety-critical, or "hard", constraints, no generalized, unified approach for guaranteeing this behavior exists. In this two-part paper, we propose an implementable set of conditions that can enforce these properties for any RTO algorithm. This second part examines the practical side of the sufficient conditions for feasibility and optimality (SCFO) proposed in the first and focuses on how they may be enforced in real application, where much of the knowledge required for the conceptual SCFO is unavailable. Methods for improving convergence speed are also considered.Comment: 56 pages, 15 figure

Sufficient Conditions for Feasibility and Optimality of Real-Time Optimization Schemes - I. Theoretical Foundations

The idea of iterative process optimization based on collected output measurements, or "real-time optimization" (RTO), has gained much prominence in recent decades, with many RTO algorithms being proposed, researched, and developed. While the essential goal of these schemes is to drive the process to its true optimal conditions without violating any safety-critical, or "hard", constraints, no generalized, unified approach for guaranteeing this behavior exists. In this two-part paper, we propose an implementable set of conditions that can enforce these properties for any RTO algorithm. The first part of the work is dedicated to the theory behind the sufficient conditions for feasibility and optimality (SCFO), together with their basic implementation strategy. RTO algorithms enforcing the SCFO are shown to perform as desired in several numerical examples - allowing for feasible-side convergence to the plant optimum where algorithms not enforcing the conditions would fail.Comment: Working paper; supplementary material available at: http://infoscience.epfl.ch/record/18807