The Separation Principle in Stochastic Control, Redux

Over the last 50 years a steady stream of accounts have been written on the separation principle of stochastic control. Even in the context of the linear-quadratic regulator in continuous time with Gaussian white noise, subtle difficulties arise, unexpected by many, that are often overlooked. In this paper we propose a new framework for establishing the separation principle. This approach takes the viewpoint that stochastic systems are well-defined maps between sample paths rather than stochastic processes per se and allows us to extend the separation principle to systems driven by martingales with possible jumps. While the approach is more in line with "real-life" engineering thinking where signals travel around the feedback loop, it is unconventional from a probabilistic point of view in that control laws for which the feedback equations are satisfied almost surely, and not deterministically for every sample path, are excluded.Comment: 23 pages, 6 figures, 2nd revision: added references, correction

Optimal Control Strategies in Delayed Sharing Information Structures

The $n$-step delayed sharing information structure is investigated. This information structure comprises of $K$ controllers that share their information with a delay of $n$ time steps. This information structure is a link between the classical information structure, where information is shared perfectly between the controllers, and a non-classical information structure, where there is no "lateral" sharing of information among the controllers. Structural results for optimal control strategies for systems with such information structures are presented. A sequential methodology for finding the optimal strategies is also derived. The solution approach provides an insight for identifying structural results and sequential decomposition for general decentralized stochastic control problems.Comment: Sumbitted to IEEE Transactions on automatic contro

Convergence Analysis of Mixed Timescale Cross-Layer Stochastic Optimization

This paper considers a cross-layer optimization problem driven by multi-timescale stochastic exogenous processes in wireless communication networks. Due to the hierarchical information structure in a wireless network, a mixed timescale stochastic iterative algorithm is proposed to track the time-varying optimal solution of the cross-layer optimization problem, where the variables are partitioned into short-term controls updated in a faster timescale, and long-term controls updated in a slower timescale. We focus on establishing a convergence analysis framework for such multi-timescale algorithms, which is difficult due to the timescale separation of the algorithm and the time-varying nature of the exogenous processes. To cope with this challenge, we model the algorithm dynamics using stochastic differential equations (SDEs) and show that the study of the algorithm convergence is equivalent to the study of the stochastic stability of a virtual stochastic dynamic system (VSDS). Leveraging the techniques of Lyapunov stability, we derive a sufficient condition for the algorithm stability and a tracking error bound in terms of the parameters of the multi-timescale exogenous processes. Based on these results, an adaptive compensation algorithm is proposed to enhance the tracking performance. Finally, we illustrate the framework by an application example in wireless heterogeneous network

Towards sub-optimal stochastic control of partially observable stochastic systems

A class of multidimensional stochastic control problems with noisy data and bounded controls encountered in aerospace design is examined. The emphasis is on suboptimal design, the optimality being taken in quadratic mean sense. To that effect the problem is viewed as a stochastic version of the Lurie problem known from nonlinear control theory. The main result is a separation theorem (involving a nonlinear Kalman-like filter) suitable for Lurie-type approximations. The theorem allows for discontinuous characteristics. As a byproduct the existence of strong solutions to a class of non-Lipschitzian stochastic differential equations in dimensions is proven