Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

On general systems with network-enhanced complexities

In recent years, the study of networked control systems (NCSs) has gradually become an active research area due to the advantages of using networked media in many aspects such as the ease of maintenance and installation, the large flexibility and the low cost. It is well known that the devices in networks are mutually connected via communication cables that are of limited capacity. Therefore, some network-induced phenomena have inevitably emerged in the areas of signal processing and control engineering. These phenomena include, but are not limited to, network-induced communication delays, missing data, signal quantization, saturations, and channel fading. It is of great importance to understand how these phenomena influence the closed-loop stability and performance properties

Adaptive-predictive control strategy for HVAC systems in smart buildings – A review

Abstract High share of energy consumption in buildings and subsequent increase in greenhouse gas emissions along with stricter legislations have motivated researchers to look for sustainable solutions in order to reduce energy consumption by using alternative renewable energy resources and improving the efficiency in this sector. Today, the smart building and socially resilient city concepts have been introduced where building automation technologies are implemented to manage and control the energy generation/consumption/storage. Building automation and control systems can be roughly classified into traditional and advanced control strategies. Traditional strategies are not a viable choice for more sophisticated features required in smart buildings. The main focus of this paper is to review advanced control strategies and their impact on buildings and technical systems with respect to energy/cost saving. These strategies should be predictive/responsive/adaptive against weather, user, grid and thermal mass. In this context, special attention is paid to model predictive control and adaptive control strategies. Although model predictive control is the most common type used in buildings, it is not well suited for systems consisting of uncertainties and unpredictable data. Thus, adaptive predictive control strategies are being developed to address these shortcomings. Despite great progress in this field, the quantified results of these strategies reported in literature showed a high level of inconsistency. This is due to the application of different control modes, various boundary conditions, hypotheses, fields of application, and type of energy consumption in different studies. Thus, this review assesses the implementations and configurations of advanced control solutions and highlights research gaps in this field that need further investigations

Event-triggered near optimal adaptive control of interconnected systems

Increased interest in complex interconnected systems like smart-grid, cyber manufacturing have attracted researchers to develop optimal adaptive control schemes to elicit a desired performance when the complex system dynamics are uncertain. In this dissertation, motivated by the fact that aperiodic event sampling saves network resources while ensuring system stability, a suite of novel event-sampled distributed near-optimal adaptive control schemes are introduced for uncertain linear and affine nonlinear interconnected systems in a forward-in-time and online manner. First, a novel stochastic hybrid Q-learning scheme is proposed to generate optimal adaptive control law and to accelerate the learning process in the presence of random delays and packet losses resulting from the communication network for an uncertain linear interconnected system. Subsequently, a novel online reinforcement learning (RL) approach is proposed to solve the Hamilton-Jacobi-Bellman (HJB) equation by using neural networks (NNs) for generating distributed optimal control of nonlinear interconnected systems using state and output feedback. To relax the state vector measurements, distributed observers are introduced. Next, using RL, an improved NN learning rule is derived to solve the HJB equation for uncertain nonlinear interconnected systems with event-triggered feedback. Distributed NN identifiers are introduced both for approximating the uncertain nonlinear dynamics and to serve as a model for online exploration. Next, the control policy and the event-sampling errors are considered as non-cooperative players and a min-max optimization problem is formulated for linear and affine nonlinear systems by using zero-sum game approach for simultaneous optimization of both the control policy and the event based sampling instants. The net result is the development of optimal adaptive event-triggered control of uncertain dynamic systems --Abstract, page iv

Real-Time Stochastic Predictive Control for Hybrid Vehicle Energy Management.

This work presents three computational methods for real time energy management in a hybrid hydraulic vehicle (HHV) when driver behavior and vehicle route are not known in advance. These methods, implemented in a receding horizon control (aka model predictive control) framework, are rather general and can be applied to systems with nonlinear dynamics subject to a Markov disturbance. State and input constraints are considered in each method. A mechanism based on the steady state distribution of the underlying Markov chain is developed for planning beyond a finite horizon in the HHV energy management problem. Road elevation information is forecasted along the horizon and then merged with the statistical model of driver behavior to increase accuracy of the horizon optimization. The characteristics of each strategy are compared and the benefit of learning driver behavior is analyzed through simulation on three drive cycles, including one real world drive cycle. A simulation is designed to explicitly demonstrate the benefit of adapting the Markov chain to real time driver behavior. Experimental results demonstrate the real time potential of the primary algorithm when implemented on a processor with limited computational resources

Event sampled optimal adaptive regulation of linear and a class of nonlinear systems

In networked control systems (NCS), wherein a communication network is used to close the feedback loop, the transmission of feedback signals and execution of the controller is currently carried out at periodic sampling instants. Thus, this scheme requires a significant computational power and network bandwidth. In contrast, the event-based aperiodic sampling and control, which is introduced recently, appears to relieve the computational burden and high network resource utilization. Therefore, in this dissertation, a suite of novel event sampled adaptive regulation schemes in both discrete and continuous time domain for uncertain linear and nonlinear systems are designed. Event sampled Q-learning and adaptive/neuro dynamic programming (ADP) schemes without value and policy iterations are utilized for the linear and nonlinear systems, respectively, in both the time domains. Neural networks (NN) are employed as approximators for nonlinear systems and, hence, the universal approximation property of NN in the event-sampled framework is introduced. The tuning of the parameters and the NN weights are carried out in an aperiodic manner at the event sampled instants leading to a further saving in computation when compared to traditional NN based control. The adaptive regulator when applied on a linear NCS with time-varying network delays and packet losses shows a 30% and 56% reduction in computation and network bandwidth usage, respectively. In case of nonlinear NCS with event sampled ADP based regulator, a reduction of 27% and 66% is observed when compared to periodic sampled schemes. The sampling and transmission instants are determined through adaptive event sampling conditions derived using Lyapunov technique by viewing the closed-loop event sampled linear and nonlinear systems as switched and/or impulsive dynamical systems. --Abstract, page iii

Computation-Communication Trade-offs and Sensor Selection in Real-time Estimation for Processing Networks

Recent advances in electronics are enabling substantial processing to be performed at each node (robots, sensors) of a networked system. Local processing enables data compression and may mitigate measurement noise, but it is still slower compared to a central computer (it entails a larger computational delay). However, while nodes can process the data in parallel, the centralized computational is sequential in nature. On the other hand, if a node sends raw data to a central computer for processing, it incurs communication delay. This leads to a fundamental communication-computation trade-off, where each node has to decide on the optimal amount of preprocessing in order to maximize the network performance. We consider a network in charge of estimating the state of a dynamical system and provide three contributions. First, we provide a rigorous problem formulation for optimal real-time estimation in processing networks in the presence of delays. Second, we show that, in the case of a homogeneous network (where all sensors have the same computation) that monitors a continuous-time scalar linear system, the optimal amount of local preprocessing maximizing the network estimation performance can be computed analytically. Third, we consider the realistic case of a heterogeneous network monitoring a discrete-time multi-variate linear system and provide algorithms to decide on suitable preprocessing at each node, and to select a sensor subset when computational constraints make using all sensors suboptimal. Numerical simulations show that selecting the sensors is crucial. Moreover, we show that if the nodes apply the preprocessing policy suggested by our algorithms, they can largely improve the network estimation performance.Comment: 15 pages, 16 figures. Accepted journal versio

The WITCH Model. Structure, Baseline, Solutions

WITCH – World Induced Technical Change Hybrid – is a regionally disaggregated hard-link hybrid global model with a neoclassical optimal growth structure (top-down) and a detailed energy input component (bottom-up). The model endogenously accounts for technological change, both through learning curves that affect the prices of new vintages of capital and through R&D investments. The model features the main economic and environmental policies in each world region as the outcome of a dynamic game. WITCH belongs to the class of Integrated Assessment Models as it possesses a climate module that feeds climate changes back into the economy. Although the model’s main features are discussed elsewhere (Bosetti et al., 2006), here we provide a more thorough discussion of the model’s structure and baseline projections, to describe the model in greater detail. We report detailed information on the evolution of energy demand, technology and CO2 emissions. We also explain the procedure used to calibrate the model parameters. This report is therefore meant to provide effective support to those who intending to use the WITCH model or interpret its results.Climate Policy, Hybrid Modelling, Integrated Assessment, Technological Change