46,185 research outputs found
Learning non-Markovian Decision-Making from State-only Sequences
Conventional imitation learning assumes access to the actions of
demonstrators, but these motor signals are often non-observable in naturalistic
settings. Additionally, sequential decision-making behaviors in these settings
can deviate from the assumptions of a standard Markov Decision Process (MDP).
To address these challenges, we explore deep generative modeling of state-only
sequences with non-Markov Decision Process (nMDP), where the policy is an
energy-based prior in the latent space of the state transition generator. We
develop maximum likelihood estimation to achieve model-based imitation, which
involves short-run MCMC sampling from the prior and importance sampling for the
posterior. The learned model enables \textit{decision-making as inference}:
model-free policy execution is equivalent to prior sampling, model-based
planning is posterior sampling initialized from the policy. We demonstrate the
efficacy of the proposed method in a prototypical path planning task with
non-Markovian constraints and show that the learned model exhibits strong
performances in challenging domains from the MuJoCo suite
Causal Confusion in Imitation Learning
Behavioral cloning reduces policy learning to supervised learning by training
a discriminative model to predict expert actions given observations. Such
discriminative models are non-causal: the training procedure is unaware of the
causal structure of the interaction between the expert and the environment. We
point out that ignoring causality is particularly damaging because of the
distributional shift in imitation learning. In particular, it leads to a
counter-intuitive "causal misidentification" phenomenon: access to more
information can yield worse performance. We investigate how this problem
arises, and propose a solution to combat it through targeted
interventions---either environment interaction or expert queries---to determine
the correct causal model. We show that causal misidentification occurs in
several benchmark control domains as well as realistic driving settings, and
validate our solution against DAgger and other baselines and ablations.Comment: Published at NeurIPS 2019 9 pages, plus references and appendice
- …