8,800 research outputs found
Value Iteration for Long-run Average Reward in Markov Decision Processes
Markov decision processes (MDPs) are standard models for probabilistic
systems with non-deterministic behaviours. Long-run average rewards provide a
mathematically elegant formalism for expressing long term performance. Value
iteration (VI) is one of the simplest and most efficient algorithmic approaches
to MDPs with other properties, such as reachability objectives. Unfortunately,
a naive extension of VI does not work for MDPs with long-run average rewards,
as there is no known stopping criterion. In this work our contributions are
threefold. (1) We refute a conjecture related to stopping criteria for MDPs
with long-run average rewards. (2) We present two practical algorithms for MDPs
with long-run average rewards based on VI. First, we show that a combination of
applying VI locally for each maximal end-component (MEC) and VI for
reachability objectives can provide approximation guarantees. Second, extending
the above approach with a simulation-guided on-demand variant of VI, we present
an anytime algorithm that is able to deal with very large models. (3) Finally,
we present experimental results showing that our methods significantly
outperform the standard approaches on several benchmarks
Magnifying Lens Abstraction for Stochastic Games with Discounted and Long-run Average Objectives
Turn-based stochastic games and its important subclass Markov decision
processes (MDPs) provide models for systems with both probabilistic and
nondeterministic behaviors. We consider turn-based stochastic games with two
classical quantitative objectives: discounted-sum and long-run average
objectives. The game models and the quantitative objectives are widely used in
probabilistic verification, planning, optimal inventory control, network
protocol and performance analysis. Games and MDPs that model realistic systems
often have very large state spaces, and probabilistic abstraction techniques
are necessary to handle the state-space explosion. The commonly used
full-abstraction techniques do not yield space-savings for systems that have
many states with similar value, but does not necessarily have similar
transition structure. A semi-abstraction technique, namely Magnifying-lens
abstractions (MLA), that clusters states based on value only, disregarding
differences in their transition relation was proposed for qualitative
objectives (reachability and safety objectives). In this paper we extend the
MLA technique to solve stochastic games with discounted-sum and long-run
average objectives. We present the MLA technique based abstraction-refinement
algorithm for stochastic games and MDPs with discounted-sum objectives. For
long-run average objectives, our solution works for all MDPs and a sub-class of
stochastic games where every state has the same value
Reinforcement Learning: A Survey
This paper surveys the field of reinforcement learning from a
computer-science perspective. It is written to be accessible to researchers
familiar with machine learning. Both the historical basis of the field and a
broad selection of current work are summarized. Reinforcement learning is the
problem faced by an agent that learns behavior through trial-and-error
interactions with a dynamic environment. The work described here has a
resemblance to work in psychology, but differs considerably in the details and
in the use of the word ``reinforcement.'' The paper discusses central issues of
reinforcement learning, including trading off exploration and exploitation,
establishing the foundations of the field via Markov decision theory, learning
from delayed reinforcement, constructing empirical models to accelerate
learning, making use of generalization and hierarchy, and coping with hidden
state. It concludes with a survey of some implemented systems and an assessment
of the practical utility of current methods for reinforcement learning.Comment: See http://www.jair.org/ for any accompanying file
- …