1,264 research outputs found
Hierarchical Decomposition of Nonlinear Dynamics and Control for System Identification and Policy Distillation
The control of nonlinear dynamical systems remains a major challenge for
autonomous agents. Current trends in reinforcement learning (RL) focus on
complex representations of dynamics and policies, which have yielded impressive
results in solving a variety of hard control tasks. However, this new
sophistication and extremely over-parameterized models have come with the cost
of an overall reduction in our ability to interpret the resulting policies. In
this paper, we take inspiration from the control community and apply the
principles of hybrid switching systems in order to break down complex dynamics
into simpler components. We exploit the rich representational power of
probabilistic graphical models and derive an expectation-maximization (EM)
algorithm for learning a sequence model to capture the temporal structure of
the data and automatically decompose nonlinear dynamics into stochastic
switching linear dynamical systems. Moreover, we show how this framework of
switching models enables extracting hierarchies of Markovian and
auto-regressive locally linear controllers from nonlinear experts in an
imitation learning scenario.Comment: 2nd Annual Conference on Learning for Dynamics and Contro
Model-Based Reinforcement Learning for Stochastic Hybrid Systems
Optimal control of general nonlinear systems is a central challenge in
automation. Enabled by powerful function approximators, data-driven approaches
to control have recently successfully tackled challenging robotic applications.
However, such methods often obscure the structure of dynamics and control
behind black-box over-parameterized representations, thus limiting our ability
to understand closed-loop behavior. This paper adopts a hybrid-system view of
nonlinear modeling and control that lends an explicit hierarchical structure to
the problem and breaks down complex dynamics into simpler localized units. We
consider a sequence modeling paradigm that captures the temporal structure of
the data and derive an expectation-maximization (EM) algorithm that
automatically decomposes nonlinear dynamics into stochastic piecewise affine
dynamical systems with nonlinear boundaries. Furthermore, we show that these
time-series models naturally admit a closed-loop extension that we use to
extract local polynomial feedback controllers from nonlinear experts via
behavioral cloning. Finally, we introduce a novel hybrid relative entropy
policy search (Hb-REPS) technique that incorporates the hierarchical nature of
hybrid systems and optimizes a set of time-invariant local feedback controllers
derived from a local polynomial approximation of a global state-value function
Variational Bayes with Intractable Likelihood
Variational Bayes (VB) is rapidly becoming a popular tool for Bayesian
inference in statistical modeling. However, the existing VB algorithms are
restricted to cases where the likelihood is tractable, which precludes the use
of VB in many interesting situations such as in state space models and in
approximate Bayesian computation (ABC), where application of VB methods was
previously impossible. This paper extends the scope of application of VB to
cases where the likelihood is intractable, but can be estimated unbiasedly. The
proposed VB method therefore makes it possible to carry out Bayesian inference
in many statistical applications, including state space models and ABC. The
method is generic in the sense that it can be applied to almost all statistical
models without requiring too much model-based derivation, which is a drawback
of many existing VB algorithms. We also show how the proposed method can be
used to obtain highly accurate VB approximations of marginal posterior
distributions.Comment: 40 pages, 6 figure
- …