2,505 research outputs found
Stochastic Shortest Path with Energy Constraints in POMDPs
We consider partially observable Markov decision processes (POMDPs) with a
set of target states and positive integer costs associated with every
transition. The traditional optimization objective (stochastic shortest path)
asks to minimize the expected total cost until the target set is reached. We
extend the traditional framework of POMDPs to model energy consumption, which
represents a hard constraint. The energy levels may increase and decrease with
transitions, and the hard constraint requires that the energy level must remain
positive in all steps till the target is reached. First, we present a novel
algorithm for solving POMDPs with energy levels, developing on existing POMDP
solvers and using RTDP as its main method. Our second contribution is related
to policy representation. For larger POMDP instances the policies computed by
existing solvers are too large to be understandable. We present an automated
procedure based on machine learning techniques that automatically extracts
important decisions of the policy allowing us to compute succinct human
readable policies. Finally, we show experimentally that our algorithm performs
well and computes succinct policies on a number of POMDP instances from the
literature that were naturally enhanced with energy levels.Comment: Technical report accompanying a paper published in proceedings of
AAMAS 201
Planning in POMDPs Using Multiplicity Automata
Planning and learning in Partially Observable MDPs (POMDPs) are among the
most challenging tasks in both the AI and Operation Research communities.
Although solutions to these problems are intractable in general, there might be
special cases, such as structured POMDPs, which can be solved efficiently. A
natural and possibly efficient way to represent a POMDP is through the
predictive state representation (PSR) - a representation which recently has
been receiving increasing attention. In this work, we relate POMDPs to
multiplicity automata- showing that POMDPs can be represented by multiplicity
automata with no increase in the representation size. Furthermore, we show that
the size of the multiplicity automaton is equal to the rank of the predictive
state representation. Therefore, we relate both the predictive state
representation and POMDPs to the well-founded multiplicity automata literature.
Based on the multiplicity automata representation, we provide a planning
algorithm which is exponential only in the multiplicity automata rank rather
than the number of states of the POMDP. As a result, whenever the predictive
state representation is logarithmic in the standard POMDP representation, our
planning algorithm is efficient.Comment: Appears in Proceedings of the Twenty-First Conference on Uncertainty
in Artificial Intelligence (UAI2005
Learning to Represent Haptic Feedback for Partially-Observable Tasks
The sense of touch, being the earliest sensory system to develop in a human
body [1], plays a critical part of our daily interaction with the environment.
In order to successfully complete a task, many manipulation interactions
require incorporating haptic feedback. However, manually designing a feedback
mechanism can be extremely challenging. In this work, we consider manipulation
tasks that need to incorporate tactile sensor feedback in order to modify a
provided nominal plan. To incorporate partial observation, we present a new
framework that models the task as a partially observable Markov decision
process (POMDP) and learns an appropriate representation of haptic feedback
which can serve as the state for a POMDP model. The model, that is parametrized
by deep recurrent neural networks, utilizes variational Bayes methods to
optimize the approximate posterior. Finally, we build on deep Q-learning to be
able to select the optimal action in each state without access to a simulator.
We test our model on a PR2 robot for multiple tasks of turning a knob until it
clicks.Comment: IEEE International Conference on Robotics and Automation (ICRA), 201
- …