716 research outputs found
Minimum Description Length Control
We propose a novel framework for multitask reinforcement learning based on
the minimum description length (MDL) principle. In this approach, which we term
MDL-control (MDL-C), the agent learns the common structure among the tasks with
which it is faced and then distills it into a simpler representation which
facilitates faster convergence and generalization to new tasks. In doing so,
MDL-C naturally balances adaptation to each task with epistemic uncertainty
about the task distribution. We motivate MDL-C via formal connections between
the MDL principle and Bayesian inference, derive theoretical performance
guarantees, and demonstrate MDL-C's empirical effectiveness on both discrete
and high-dimensional continuous control tasks
Making Risk Minimization Tolerant to Label Noise
In many applications, the training data, from which one needs to learn a
classifier, is corrupted with label noise. Many standard algorithms such as SVM
perform poorly in presence of label noise. In this paper we investigate the
robustness of risk minimization to label noise. We prove a sufficient condition
on a loss function for the risk minimization under that loss to be tolerant to
uniform label noise. We show that the loss, sigmoid loss, ramp loss and
probit loss satisfy this condition though none of the standard convex loss
functions satisfy it. We also prove that, by choosing a sufficiently large
value of a parameter in the loss function, the sigmoid loss, ramp loss and
probit loss can be made tolerant to non-uniform label noise also if we can
assume the classes to be separable under noise-free data distribution. Through
extensive empirical studies, we show that risk minimization under the
loss, the sigmoid loss and the ramp loss has much better robustness to label
noise when compared to the SVM algorithm
Recovery Guarantees for Quadratic Tensors with Limited Observations
We consider the tensor completion problem of predicting the missing entries
of a tensor. The commonly used CP model has a triple product form, but an
alternate family of quadratic models which are the sum of pairwise products
instead of a triple product have emerged from applications such as
recommendation systems. Non-convex methods are the method of choice for
learning quadratic models, and this work examines their sample complexity and
error guarantee. Our main result is that with the number of samples being only
linear in the dimension, all local minima of the mean squared error objective
are global minima and recover the original tensor accurately. The techniques
lead to simple proofs showing that convex relaxation can recover quadratic
tensors provided with linear number of samples. We substantiate our theoretical
results with experiments on synthetic and real-world data, showing that
quadratic models have better performance than CP models in scenarios where
there are limited amount of observations available
Reinforcement Learning Your Way : Agent Characterization through Policy Regularization
The increased complexity of state-of-the-art reinforcement learning (RL) algorithms has resulted in an opacity that inhibits explainability and understanding. This has led to the development of several post hoc explainability methods that aim to extract information from learned policies, thus aiding explainability. These methods rely on empirical observations of the policy, and thus aim to generalize a characterization of agents’ behaviour. In this study, we have instead developed a method to imbue agents’ policies with a characteristic behaviour through regularization of their objective functions. Our method guides the agents’ behaviour during learning, which results in an intrinsic characterization; it connects the learning process with model explanation. We provide a formal argument and empirical evidence for the viability of our method. In future work, we intend to employ it to develop agents that optimize individual financial customers’ investment portfolios based on their spending personalities.publishedVersio
On Pathologies in KL-Regularized Reinforcement Learning from Expert Demonstrations
KL-regularized reinforcement learning from expert demonstrations has proved
successful in improving the sample efficiency of deep reinforcement learning
algorithms, allowing them to be applied to challenging physical real-world
tasks. However, we show that KL-regularized reinforcement learning with
behavioral reference policies derived from expert demonstrations can suffer
from pathological training dynamics that can lead to slow, unstable, and
suboptimal online learning. We show empirically that the pathology occurs for
commonly chosen behavioral policy classes and demonstrate its impact on sample
efficiency and online policy performance. Finally, we show that the pathology
can be remedied by non-parametric behavioral reference policies and that this
allows KL-regularized reinforcement learning to significantly outperform
state-of-the-art approaches on a variety of challenging locomotion and
dexterous hand manipulation tasks.Comment: Published in Advances in Neural Information Processing Systems 34
(NeurIPS 2021
- …