51 research outputs found

    Optimizing Memory-Bounded Controllers for Decentralized POMDPs

    Full text link
    We present a memory-bounded optimization approach for solving infinite-horizon decentralized POMDPs. Policies for each agent are represented by stochastic finite state controllers. We formulate the problem of optimizing these policies as a nonlinear program, leveraging powerful existing nonlinear optimization techniques for solving the problem. While existing solvers only guarantee locally optimal solutions, we show that our formulation produces higher quality controllers than the state-of-the-art approach. We also incorporate a shared source of randomness in the form of a correlation device to further increase solution quality with only a limited increase in space and time. Our experimental results show that nonlinear optimization can be used to provide high quality, concise solutions to decentralized decision problems under uncertainty.Comment: Appears in Proceedings of the Twenty-Third Conference on Uncertainty in Artificial Intelligence (UAI2007

    On the Computational Complexity of Stochastic Controller Optimization in POMDPs

    Get PDF
    We show that the problem of finding an optimal stochastic 'blind' controller in a Markov decision process is an NP-hard problem. The corresponding decision problem is NP-hard, in PSPACE, and SQRT-SUM-hard, hence placing it in NP would imply breakthroughs in long-standing open problems in computer science. Our result establishes that the more general problem of stochastic controller optimization in POMDPs is also NP-hard. Nonetheless, we outline a special case that is convex and admits efficient global solutions.Comment: Corrected error in the proof of Theorem 2, and revised Section

    Stochastic Finite State Control of POMDPs with LTL Specifications

    Get PDF
    Partially observable Markov decision processes (POMDPs) provide a modeling framework for autonomous decision making under uncertainty and imperfect sensing, e.g. robot manipulation and self-driving cars. However, optimal control of POMDPs is notoriously intractable. This paper considers the quantitative problem of synthesizing sub-optimal stochastic finite state controllers (sFSCs) for POMDPs such that the probability of satisfying a set of high-level specifications in terms of linear temporal logic (LTL) formulae is maximized. We begin by casting the latter problem into an optimization and use relaxations based on the Poisson equation and McCormick envelopes. Then, we propose an stochastic bounded policy iteration algorithm, leading to a controlled growth in sFSC size and an any time algorithm, where the performance of the controller improves with successive iterations, but can be stopped by the user based on time or memory considerations. We illustrate the proposed method by a robot navigation case study

    History-Based Controller Design and Optimization for Partially Observable MDPs

    Get PDF
    Partially observable MDPs provide an elegant framework for sequential decision making. Finite-state controllers (FSCs) are often used to represent policies for infinite-horizon prob-lems as they offer a compact representation, simple-to-execute plans, and adjustable tradeoff between computational complexity and policy size. We develop novel connections between optimizing FSCs for POMDPs and the dual linear program for MDPs. Building on that, we present a dual mixed integer linear program (MIP) for optimizing FSCs. To assign well-defined meaning to FSC nodes as well as aid in policy search, we show how to associate history-based features with each FSC node. Using this representation, we address an-other challenging problem, that of iteratively deciding which nodes to add to FSC to get a better policy. Using an efficient off-the-shelf MIP solver, we show that this new approach can find compact near-optimal FSCs for several large benchmark domains, and is competitive with previous best approaches.

    Parameter Synthesis in Markov Models: A Gentle Survey

    Full text link
    This paper surveys the analysis of parametric Markov models whose transitions are labelled with functions over a finite set of parameters. These models are symbolic representations of uncountable many concrete probabilistic models, each obtained by instantiating the parameters. We consider various analysis problems for a given logical specification φ\varphi: do all parameter instantiations within a given region of parameter values satisfy φ\varphi?, which instantiations satisfy φ\varphi and which ones do not?, and how can all such instantiations be characterised, either exactly or approximately? We address theoretical complexity results and describe the main ideas underlying state-of-the-art algorithms that established an impressive leap over the last decade enabling the fully automated analysis of models with millions of states and thousands of parameters

    Stochastic and Optimal Distributed Control for Energy Optimization and Spatially Invariant Systems

    Get PDF
    Improving energy efficiency and grid responsiveness of buildings requires sensing, computing and communication to enable stochastic decision-making and distributed operations. Optimal control synthesis plays a significant role in dealing with the complexity and uncertainty associated with the energy systems. The dissertation studies general area of complex networked systems that consist of interconnected components and usually operate in uncertain environments. Specifically, the contents of this dissertation include tools using stochastic and optimal distributed control to overcome these challenges and improve the sustainability of electric energy systems. The first tool is developed as a unifying stochastic control approach for improving energy efficiency while meeting probabilistic constraints. This algorithm is applied to demonstrate energy efficiency improvement in buildings and improving operational efficiency of virtualized web servers, respectively. Although all the optimization in this technique is in the form of convex optimization, it heavily relies on semidefinite programming (SP). A generic SP solver can handle only up to hundreds of variables. This being said, for a large scale system, the existing off-the-shelf algorithms may not be an appropriate tool for optimal control. Therefore, in the sequel I will exploit optimization in a distributed way. The second tool is itself a concrete study which is optimal distributed control for spatially invariant systems. Spatially invariance means the dynamics of the system do not vary as we translate along some spatial axis. The optimal H2 [H-2] decentralized control problem is solved by computing an orthogonal projection on a class of Youla parameters with a decentralized structure. Optimal H∞ [H-infinity] performance is posed as a distance minimization in a general L∞ [L-infinity] space from a vector function to a subspace with a mixed L∞ and H∞ space structure. In this framework, the dual and pre-dual formulations lead to finite dimensional convex optimizations which approximate the optimal solution within desired accuracy. Furthermore, a mixed L2 [L-2] /H∞ synthesis problem for spatially invariant systems as trade-offs between transient performance and robustness. Finally, we pursue to deal with a more general networked system, i.e. the Non-Markovian decentralized stochastic control problem, using stochastic maximum principle via Malliavin Calculus
    • …
    corecore