788 research outputs found
Human Apprenticeship Learning via Kernel-based Inverse Reinforcement Learning
It has been well demonstrated that inverse reinforcement learning (IRL) is an
effective technique for teaching machines to perform tasks at human skill
levels given human demonstrations (i.e., human to machine apprenticeship
learning). This paper seeks to show that a similar application can be
demonstrated with human learners. That is, given demonstrations from human
experts inverse reinforcement learning techniques can be used to teach other
humans to perform at higher skill levels (i.e., human to human apprenticeship
learning). To show this two experiments were conducted using a simple,
real-time web game where players were asked to touch targets in order to earn
as many points as possible. For the experiment player performance was defined
as the number of targets a player touched, irrespective of the points that a
player actually earned. This allowed for in-game points to be modified and the
effect of these alterations on performance measured. At no time were
participants told the true performance metric. To determine the point
modifications IRL was applied on demonstrations of human experts playing the
game. The results of the experiment show with significance that performance
improved over the control for select treatment groups. Finally, in addition to
the experiment, we also detail the algorithmic challenges we faced when
conducting the experiment and the techniques we used to overcome them.Comment: 31 pages, 23 figures, Submitted to Journal of Artificial Intelligence
Research, "for source code, see https://github.com/mrucker/kpirl-kla
Inverse Reinforcement Learning in Large State Spaces via Function Approximation
This paper introduces a new method for inverse reinforcement learning in
large-scale and high-dimensional state spaces. To avoid solving the
computationally expensive reinforcement learning problems in reward learning,
we propose a function approximation method to ensure that the Bellman
Optimality Equation always holds, and then estimate a function to maximize the
likelihood of the observed motion. The time complexity of the proposed method
is linearly proportional to the cardinality of the action set, thus it can
handle large state spaces efficiently. We test the proposed method in a
simulated environment, and show that it is more accurate than existing methods
and significantly better in scalability. We also show that the proposed method
can extend many existing methods to high-dimensional state spaces. We then
apply the method to evaluating the effect of rehabilitative stimulations on
patients with spinal cord injuries based on the observed patient motions.Comment: Experiment update
Probabilistic inverse reinforcement learning in unknown environments
We consider the problem of learning by demonstration from agents acting in
unknown stochastic Markov environments or games. Our aim is to estimate agent
preferences in order to construct improved policies for the same task that the
agents are trying to solve. To do so, we extend previous probabilistic
approaches for inverse reinforcement learning in known MDPs to the case of
unknown dynamics or opponents. We do this by deriving two simplified
probabilistic models of the demonstrator's policy and utility. For
tractability, we use maximum a posteriori estimation rather than full Bayesian
inference. Under a flat prior, this results in a convex optimisation problem.
We find that the resulting algorithms are highly competitive against a variety
of other methods for inverse reinforcement learning that do have knowledge of
the dynamics.Comment: Appears in Proceedings of the Twenty-Ninth Conference on Uncertainty
in Artificial Intelligence (UAI2013
Human-Machine Collaborative Optimization via Apprenticeship Scheduling
Coordinating agents to complete a set of tasks with intercoupled temporal and
resource constraints is computationally challenging, yet human domain experts
can solve these difficult scheduling problems using paradigms learned through
years of apprenticeship. A process for manually codifying this domain knowledge
within a computational framework is necessary to scale beyond the
``single-expert, single-trainee" apprenticeship model. However, human domain
experts often have difficulty describing their decision-making processes,
causing the codification of this knowledge to become laborious. We propose a
new approach for capturing domain-expert heuristics through a pairwise ranking
formulation. Our approach is model-free and does not require enumerating or
iterating through a large state space. We empirically demonstrate that this
approach accurately learns multifaceted heuristics on a synthetic data set
incorporating job-shop scheduling and vehicle routing problems, as well as on
two real-world data sets consisting of demonstrations of experts solving a
weapon-to-target assignment problem and a hospital resource allocation problem.
We also demonstrate that policies learned from human scheduling demonstration
via apprenticeship learning can substantially improve the efficiency of a
branch-and-bound search for an optimal schedule. We employ this human-machine
collaborative optimization technique on a variant of the weapon-to-target
assignment problem. We demonstrate that this technique generates solutions
substantially superior to those produced by human domain experts at a rate up
to 9.5 times faster than an optimization approach and can be applied to
optimally solve problems twice as complex as those solved by a human
demonstrator.Comment: Portions of this paper were published in the Proceedings of the
International Joint Conference on Artificial Intelligence (IJCAI) in 2016 and
in the Proceedings of Robotics: Science and Systems (RSS) in 2016. The paper
consists of 50 pages with 11 figures and 4 table
- …