Learning Throttle Valve Control Using Policy Search

Abstract. The throttle valve is a technical device used for regulating a fluid or a gas flow. Throttle valve control is a challenging task, due to its complex dynamics and demanding constraints for the controller. Using state-of-the-art throttle valve control, such as model-free PID controllers, time-consuming and manual adjusting of the controller is necessary. In this paper, we investigate how reinforcement learning (RL) can help to alleviate the effort of manual controller design by automatically learning a control policy from experiences. In order to obtain a valid control policy for the throttle valve, several constraints need to be addressed, such as no-overshoot. Furthermore, the learned controller must be able to follow given desired trajectories, while moving the valve from any start to any goal position and, thus, multi-targets policy learning needs to be considered for RL. In this study, we employ a policy search RL approach, Pilco [2], to learn a throttle valve control policy. We adapt the Pilco algorithm, while taking into account the practical requirements and constraints for the controller. For evaluation, we employ the resulting algorithm to solve several control tasks in simulation, as well as on a physical throttle valve system. The results show that policy search RL is able to learn a consistent control policy for complex, real-world systems.

Model-Based Policy Search for Automatic Tuning of Multivariate PID Controllers

PID control architectures are widely used in industrial applications. Despite their low number of open parameters, tuning multiple, coupled PID controllers can become tedious in practice. In this paper, we extend PILCO, a model-based policy search framework, to automatically tune multivariate PID controllers purely based on data observed on an otherwise unknown system. The system's state is extended appropriately to frame the PID policy as a static state feedback policy. This renders PID tuning possible as the solution of a finite horizon optimal control problem without further a priori knowledge. The framework is applied to the task of balancing an inverted pendulum on a seven degree-of-freedom robotic arm, thereby demonstrating its capabilities of fast and data-efficient policy learning, even on complex real world problems.Comment: Accepted final version to appear in 2017 IEEE International Conference on Robotics and Automation (ICRA

Economics, Control Theory, and the Phillips Machine

Can the same mathematical control laws that smooth out oscillations in the flight of an airplane also moderate economic cycles of boom and bust? Attempts to bring together the intellectual traditions of control engineering and economics go back at least as far as the hydraulic analog computer of A. W. H. Phillips, circa 1950. Today, economic policymakers remain committed to the ideal of controlling business cycles; it remains an open question whether tools from control theory might help to refine their strategies.

Flexible and robust control of heavy duty diesel engine airpath using data driven disturbance observers and GPR models

Diesel engine airpath control is crucial for modern engine development due to increasingly stringent emission regulations. This thesis aims to develop and validate a exible and robust control approach to this problem for speci cally heavy-duty engines. It focuses on estimation and control algorithms that are implementable to the current and next generation commercial electronic control units (ECU). To this end, targeting the control units in service, a data driven disturbance observer (DOB) is developed and applied for mass air ow (MAF) and manifold absolute pressure (MAP) tracking control via exhaust gas recirculation (EGR) valve and variable geometry turbine (VGT) vane. Its performance bene ts are demonstrated on the physical engine model for concept evaluation. The proposed DOB integrated with a discrete-time sliding mode controller is applied to the serial level engine control unit. Real engine performance is validated with the legal emission test cycle (WHTC - World Harmonized Transient Cycle) for heavy-duty engines and comparison with a commercially available controller is performed, and far better tracking results are obtained. Further studies are conducted in order to utilize capabilities of the next generation control units. Gaussian process regression (GPR) models are popular in automotive industry especially for emissions modeling but have not found widespread applications in airpath control yet. This thesis presents a GPR modeling of diesel engine airpath components as well as controller designs and their applications based on the developed models. Proposed GPR based feedforward and feedback controllers are validated with available physical engine models and the results have been very promisin

Hybrid IWOPSO optimization based marine engine rotational speed control automatic system

Transporting industry is having an important influence on the nations’ progress. The states that having long shoreline are taking advantage of their locations in using sea transportations which is more economical than other types of transportations. One of the most economical marine transport is the diesel fueling engines. This paper is to optimize the PID controller to control the speed of the engine overcoming the navigation environmental changes such as waves, winds and other effective external factors as well as the vessel internal changes such as the shipment load, equipment’s conditions …etc. PID is optimized through the optimum selection of its parameters (KP, KI and KD). A Simulink/MatLab model of the system is designed for this purpose. The Hybrid IWOPSO (HIWOPSO) algorithm is used for finding the optimum values of the PID parameters. The engine step response with these parameters is compared to the responses with those obtained by the IWO and PSO besides the Fuzzy Logic Control (FLC)

Effect of Atmospheric Pressure and Temperature on a Small Spark Ignition Internal Combustion Engine\u27s Performance

The ever increasing use of man portable unmanned aerial vehicles, UAV, by the US military in a wide array of environmental conditions calls for the investigation of engine performance under these conditions. Previous research has focused on individual changes in pressure or temperature conditions of the air stream entering the engine. The need was seen for a facility capable of providing an environment representative of various simulated altitude conditions. A mobile test facility was developed to test small internal combustion engines with peak powers less than 10 hp. A representative engine was tested over a range of speeds from 2000 RPM to 9000 RPM at every 1000 RPM. The throttle was set to 50%, 75%, and 100% open at each of the speeds tested. The test engine was tested at environmental conditions representing sea level standard day conditions, 1500 m conditions and 3000 m conditions. The engine torque, fuel flow rate, and air flow rate were measured at each test point to determine the impact of combined pressure and temperature variations on engine performance. During the process of testing the engine and the test stand it was determined that the fuel to air ratio for the engine had a significant impact on engine operation. The test engine failed to under fuel rich or fuel lean conditions

Output Feedback Speed Control for a Wankel Rotary Engine via Q-Learning

This paper develops a dynamic output feedback controller based on continuous-time Q-learning for the engine speed regulation problem. The proposed controller is able to learn the optimal control solution online in a finite time using only the measurable outputs. We first present the mean value engine model (MVEM) for a Wankel rotary engine. The regulation of engine speed can be formulated as an optimal control problem that minimises a pre-defined value function by actuating the electronic throttle. By parameterising an action-dependent Q-function, we derive a full-state adaptive optimal feedback controller using the idea of continuous-time Q-learning. The adaptive critic approximates the Q-function as a neural network and directly updates the actor, where the convergence is guaranteed by employing novel finite-time adaptation techniques. Then, we incorporate the extended Kalman filter (EKF) as an optimal reduced-order state observer, which enables the online estimation of the unknown fuel puddle dynamics, to achieve a dynamic output feedback engine speed controller. The simulation results of a benchmark 225CS engine demonstrate that the proposed controller can effectively regulate the engine speed to a set point under certain load disturbances

Data-Efficient Reinforcement Learning with Probabilistic Model Predictive Control

Trial-and-error based reinforcement learning (RL) has seen rapid advancements in recent times, especially with the advent of deep neural networks. However, the majority of autonomous RL algorithms require a large number of interactions with the environment. A large number of interactions may be impractical in many real-world applications, such as robotics, and many practical systems have to obey limitations in the form of state space or control constraints. To reduce the number of system interactions while simultaneously handling constraints, we propose a model-based RL framework based on probabilistic Model Predictive Control (MPC). In particular, we propose to learn a probabilistic transition model using Gaussian Processes (GPs) to incorporate model uncertainty into long-term predictions, thereby, reducing the impact of model errors. We then use MPC to find a control sequence that minimises the expected long-term cost. We provide theoretical guarantees for first-order optimality in the GP-based transition models with deterministic approximate inference for long-term planning. We demonstrate that our approach does not only achieve state-of-the-art data efficiency, but also is a principled way for RL in constrained environments.Comment: Accepted at AISTATS 2018