587 research outputs found
Feature Markov Decision Processes
General purpose intelligent learning agents cycle through (complex,non-MDP)
sequences of observations, actions, and rewards. On the other hand,
reinforcement learning is well-developed for small finite state Markov Decision
Processes (MDPs). So far it is an art performed by human designers to extract
the right state representation out of the bare observations, i.e. to reduce the
agent setup to the MDP framework. Before we can think of mechanizing this
search for suitable MDPs, we need a formal objective criterion. The main
contribution of this article is to develop such a criterion. I also integrate
the various parts into one learning algorithm. Extensions to more realistic
dynamic Bayesian networks are developed in a companion article.Comment: 7 page
Feature Reinforcement Learning: Part I: Unstructured MDPs
General-purpose, intelligent, learning agents cycle through sequences of
observations, actions, and rewards that are complex, uncertain, unknown, and
non-Markovian. On the other hand, reinforcement learning is well-developed for
small finite state Markov decision processes (MDPs). Up to now, extracting the
right state representations out of bare observations, that is, reducing the
general agent setup to the MDP framework, is an art that involves significant
effort by designers. The primary goal of this work is to automate the reduction
process and thereby significantly expand the scope of many existing
reinforcement learning algorithms and the agents that employ them. Before we
can think of mechanizing this search for suitable MDPs, we need a formal
objective criterion. The main contribution of this article is to develop such a
criterion. I also integrate the various parts into one learning algorithm.
Extensions to more realistic dynamic Bayesian networks are developed in Part
II. The role of POMDPs is also considered there.Comment: 24 LaTeX pages, 5 diagram
A Survey of Knowledge-based Sequential Decision Making under Uncertainty
Reasoning with declarative knowledge (RDK) and sequential decision-making
(SDM) are two key research areas in artificial intelligence. RDK methods reason
with declarative domain knowledge, including commonsense knowledge, that is
either provided a priori or acquired over time, while SDM methods
(probabilistic planning and reinforcement learning) seek to compute action
policies that maximize the expected cumulative utility over a time horizon;
both classes of methods reason in the presence of uncertainty. Despite the rich
literature in these two areas, researchers have not fully explored their
complementary strengths. In this paper, we survey algorithms that leverage RDK
methods while making sequential decisions under uncertainty. We discuss
significant developments, open problems, and directions for future work
Computational Approaches for Stochastic Shortest Path on Succinct MDPs
We consider the stochastic shortest path (SSP) problem for succinct Markov
decision processes (MDPs), where the MDP consists of a set of variables, and a
set of nondeterministic rules that update the variables. First, we show that
several examples from the AI literature can be modeled as succinct MDPs. Then
we present computational approaches for upper and lower bounds for the SSP
problem: (a)~for computing upper bounds, our method is polynomial-time in the
implicit description of the MDP; (b)~for lower bounds, we present a
polynomial-time (in the size of the implicit description) reduction to
quadratic programming. Our approach is applicable even to infinite-state MDPs.
Finally, we present experimental results to demonstrate the effectiveness of
our approach on several classical examples from the AI literature
A Review of Symbolic, Subsymbolic and Hybrid Methods for Sequential Decision Making
The field of Sequential Decision Making (SDM) provides tools for solving
Sequential Decision Processes (SDPs), where an agent must make a series of
decisions in order to complete a task or achieve a goal. Historically, two
competing SDM paradigms have view for supremacy. Automated Planning (AP)
proposes to solve SDPs by performing a reasoning process over a model of the
world, often represented symbolically. Conversely, Reinforcement Learning (RL)
proposes to learn the solution of the SDP from data, without a world model, and
represent the learned knowledge subsymbolically. In the spirit of
reconciliation, we provide a review of symbolic, subsymbolic and hybrid methods
for SDM. We cover both methods for solving SDPs (e.g., AP, RL and techniques
that learn to plan) and for learning aspects of their structure (e.g., world
models, state invariants and landmarks). To the best of our knowledge, no other
review in the field provides the same scope. As an additional contribution, we
discuss what properties an ideal method for SDM should exhibit and argue that
neurosymbolic AI is the current approach which most closely resembles this
ideal method. Finally, we outline several proposals to advance the field of SDM
via the integration of symbolic and subsymbolic AI
Solving Factored MDPs with Hybrid State and Action Variables
Efficient representations and solutions for large decision problems with
continuous and discrete variables are among the most important challenges faced
by the designers of automated decision support systems. In this paper, we
describe a novel hybrid factored Markov decision process (MDP) model that
allows for a compact representation of these problems, and a new hybrid
approximate linear programming (HALP) framework that permits their efficient
solutions. The central idea of HALP is to approximate the optimal value
function by a linear combination of basis functions and optimize its weights by
linear programming. We analyze both theoretical and computational aspects of
this approach, and demonstrate its scale-up potential on several hybrid
optimization problems
The 2014 International Planning Competition: Progress and Trends
We review the 2014 International Planning Competition (IPC-2014), the eighth
in a series of competitions starting in 1998. IPC-2014 was held in three separate
parts to assess state-of-the-art in three prominent areas of planning research: the
deterministic (classical) part (IPCD), the learning part (IPCL), and the probabilistic
part (IPPC). Each part evaluated planning systems in ways that pushed the edge of
existing planner performance by introducing new challenges, novel tasks, or both.
The competition surpassed again the number of competitors than its predecessor,
highlighting the competition’s central role in shaping the landscape of ongoing
developments in evaluating planning systems
- …