15 research outputs found
Hilbert Space Embeddings of POMDPs
A nonparametric approach for policy learning for POMDPs is proposed. The
approach represents distributions over the states, observations, and actions as
embeddings in feature spaces, which are reproducing kernel Hilbert spaces.
Distributions over states given the observations are obtained by applying the
kernel Bayes' rule to these distribution embeddings. Policies and value
functions are defined on the feature space over states, which leads to a
feature space expression for the Bellman equation. Value iteration may then be
used to estimate the optimal value function and associated policy. Experimental
results confirm that the correct policy is learned using the feature space
representation.Comment: Appears in Proceedings of the Twenty-Eighth Conference on Uncertainty
in Artificial Intelligence (UAI2012
A New Distribution-Free Concept for Representing, Comparing, and Propagating Uncertainty in Dynamical Systems with Kernel Probabilistic Programming
This work presents the concept of kernel mean embedding and kernel
probabilistic programming in the context of stochastic systems. We propose
formulations to represent, compare, and propagate uncertainties for fairly
general stochastic dynamics in a distribution-free manner. The new tools enjoy
sound theory rooted in functional analysis and wide applicability as
demonstrated in distinct numerical examples. The implication of this new
concept is a new mode of thinking about the statistical nature of uncertainty
in dynamical systems
Learning of non-parametric control policies with high-dimensional state features
Learning complex control policies from highdimensional sensory input is a challenge for
reinforcement learning algorithms. Kernel methods that approximate values functions
or transition models can address this problem. Yet, many current approaches rely on
instable greedy maximization. In this paper, we develop a policy search algorithm that
integrates robust policy updates and kernel embeddings. Our method can learn nonparametric
control policies for infinite horizon continuous MDPs with high-dimensional
sensory representations. We show that our method outperforms related approaches, and
that our algorithm can learn an underpowered swing-up task task directly from highdimensional
image data
Characteristic Kernels and Infinitely Divisible Distributions
We connect shift-invariant characteristic kernels to infinitely divisible
distributions on . Characteristic kernels play an important
role in machine learning applications with their kernel means to distinguish
any two probability measures. The contribution of this paper is two-fold.
First, we show, using the L\'evy-Khintchine formula, that any shift-invariant
kernel given by a bounded, continuous and symmetric probability density
function (pdf) of an infinitely divisible distribution on is
characteristic. We also present some closure property of such characteristic
kernels under addition, pointwise product, and convolution. Second, in
developing various kernel mean algorithms, it is fundamental to compute the
following values: (i) kernel mean values , , and
(ii) kernel mean RKHS inner products , for probability measures . If , and
kernel are Gaussians, then computation (i) and (ii) results in Gaussian
pdfs that is tractable. We generalize this Gaussian combination to more general
cases in the class of infinitely divisible distributions. We then introduce a
{\it conjugate} kernel and {\it convolution trick}, so that the above (i) and
(ii) have the same pdf form, expecting tractable computation at least in some
cases. As specific instances, we explore -stable distributions and a
rich class of generalized hyperbolic distributions, where the Laplace, Cauchy
and Student-t distributions are included
Optimal Rates for Regularized Conditional Mean Embedding Learning
We address the consistency of a kernel ridge regression estimate of the
conditional mean embedding (CME), which is an embedding of the conditional
distribution of given into a target reproducing kernel Hilbert space
. The CME allows us to take conditional expectations of target
RKHS functions, and has been employed in nonparametric causal and Bayesian
inference. We address the misspecified setting, where the target CME is in the
space of Hilbert-Schmidt operators acting from an input interpolation space
between and , to . This space of operators
is shown to be isomorphic to a newly defined vector-valued interpolation space.
Using this isomorphism, we derive a novel and adaptive statistical learning
rate for the empirical CME estimator under the misspecified setting. Our
analysis reveals that our rates match the optimal rates without
assuming to be finite dimensional. We further establish a lower
bound on the learning rate, which shows that the obtained upper bound is
optimal
Model-based kernel sum rule: kernel Bayesian inference with probabilistic model
Kernel Bayesian inference is a principled approach to nonparametric inference in probabilistic graphical models, where probabilistic relationships between variables are learned from data in a nonparametric manner. Various algorithms of kernel Bayesian inference have been developed by combining kernelized basic probabilistic operations such as the kernel sum rule and kernel Bayes’ rule. However, the current framework is fully nonparametric, and it does not allow a user to flexibly combine nonparametric and model-based inferences. This is inefficient when there are good probabilistic models (or simulation models) available for some parts of a graphical model; this is in particular true in scientific fields where “models” are the central topic of study. Our contribution in this paper is to introduce a novel approach, termed the model-based kernel sum rule (Mb-KSR), to combine a probabilistic model and kernel Bayesian inference. By combining the Mb-KSR with the existing kernelized probabilistic rules, one can develop various algorithms for hybrid (i.e., nonparametric and model-based) inferences. As an illustrative example, we consider Bayesian filtering in a state space model, where typically there exists an accurate probabilistic model for the state transition process. We propose a novel filtering method that combines model-based inference for the state transition process and data-driven, nonparametric inference for the observation generating process. We empirically validate our approach with synthetic and real-data experiments, the latter being the problem of vision-based mobile robot localization in robotics, which illustrates the effectiveness of the proposed hybrid approach