11,065 research outputs found
From Infinite to Finite Programs: Explicit Error Bounds with Applications to Approximate Dynamic Programming
We consider linear programming (LP) problems in infinite dimensional spaces
that are in general computationally intractable. Under suitable assumptions, we
develop an approximation bridge from the infinite-dimensional LP to tractable
finite convex programs in which the performance of the approximation is
quantified explicitly. To this end, we adopt the recent developments in two
areas of randomized optimization and first order methods, leading to a priori
as well as a posterior performance guarantees. We illustrate the generality and
implications of our theoretical results in the special case of the long-run
average cost and discounted cost optimal control problems for Markov decision
processes on Borel spaces. The applicability of the theoretical results is
demonstrated through a constrained linear quadratic optimal control problem and
a fisheries management problem.Comment: 30 pages, 5 figure
Mean-Payoff Optimization in Continuous-Time Markov Chains with Parametric Alarms
Continuous-time Markov chains with alarms (ACTMCs) allow for alarm events
that can be non-exponentially distributed. Within parametric ACTMCs, the
parameters of alarm-event distributions are not given explicitly and can be
subject of parameter synthesis. An algorithm solving the -optimal
parameter synthesis problem for parametric ACTMCs with long-run average
optimization objectives is presented. Our approach is based on reduction of the
problem to finding long-run average optimal strategies in semi-Markov decision
processes (semi-MDPs) and sufficient discretization of parameter (i.e., action)
space. Since the set of actions in the discretized semi-MDP can be very large,
a straightforward approach based on explicit action-space construction fails to
solve even simple instances of the problem. The presented algorithm uses an
enhanced policy iteration on symbolic representations of the action space. The
soundness of the algorithm is established for parametric ACTMCs with
alarm-event distributions satisfying four mild assumptions that are shown to
hold for uniform, Dirac and Weibull distributions in particular, but are
satisfied for many other distributions as well. An experimental implementation
shows that the symbolic technique substantially improves the efficiency of the
synthesis algorithm and allows to solve instances of realistic size.Comment: This article is a full version of a paper accepted to the Conference
on Quantitative Evaluation of SysTems (QEST) 201
Magnifying Lens Abstraction for Stochastic Games with Discounted and Long-run Average Objectives
Turn-based stochastic games and its important subclass Markov decision
processes (MDPs) provide models for systems with both probabilistic and
nondeterministic behaviors. We consider turn-based stochastic games with two
classical quantitative objectives: discounted-sum and long-run average
objectives. The game models and the quantitative objectives are widely used in
probabilistic verification, planning, optimal inventory control, network
protocol and performance analysis. Games and MDPs that model realistic systems
often have very large state spaces, and probabilistic abstraction techniques
are necessary to handle the state-space explosion. The commonly used
full-abstraction techniques do not yield space-savings for systems that have
many states with similar value, but does not necessarily have similar
transition structure. A semi-abstraction technique, namely Magnifying-lens
abstractions (MLA), that clusters states based on value only, disregarding
differences in their transition relation was proposed for qualitative
objectives (reachability and safety objectives). In this paper we extend the
MLA technique to solve stochastic games with discounted-sum and long-run
average objectives. We present the MLA technique based abstraction-refinement
algorithm for stochastic games and MDPs with discounted-sum objectives. For
long-run average objectives, our solution works for all MDPs and a sub-class of
stochastic games where every state has the same value
Regularized Decomposition of High-Dimensional Multistage Stochastic Programs with Markov Uncertainty
We develop a quadratic regularization approach for the solution of
high-dimensional multistage stochastic optimization problems characterized by a
potentially large number of time periods/stages (e.g. hundreds), a
high-dimensional resource state variable, and a Markov information process. The
resulting algorithms are shown to converge to an optimal policy after a finite
number of iterations under mild technical assumptions. Computational
experiments are conducted using the setting of optimizing energy storage over a
large transmission grid, which motivates both the spatial and temporal
dimensions of our problem. Our numerical results indicate that the proposed
methods exhibit significantly faster convergence than their classical
counterparts, with greater gains observed for higher-dimensional problems
Scalable First-Order Methods for Robust MDPs
Robust Markov Decision Processes (MDPs) are a powerful framework for modeling
sequential decision-making problems with model uncertainty. This paper proposes
the first first-order framework for solving robust MDPs. Our algorithm
interleaves primal-dual first-order updates with approximate Value Iteration
updates. By carefully controlling the tradeoff between the accuracy and cost of
Value Iteration updates, we achieve an ergodic convergence rate of for the best
choice of parameters on ellipsoidal and Kullback-Leibler -rectangular
uncertainty sets, where and is the number of states and actions,
respectively. Our dependence on the number of states and actions is
significantly better (by a factor of ) than that of pure
Value Iteration algorithms. In numerical experiments on ellipsoidal uncertainty
sets we show that our algorithm is significantly more scalable than
state-of-the-art approaches. Our framework is also the first one to solve
robust MDPs with -rectangular KL uncertainty sets
- …