67 research outputs found
Wasserstein Distance guided Adversarial Imitation Learning with Reward Shape Exploration
The generative adversarial imitation learning (GAIL) has provided an
adversarial learning framework for imitating expert policy from demonstrations
in high-dimensional continuous tasks. However, almost all GAIL and its
extensions only design a kind of reward function of logarithmic form in the
adversarial training strategy with the Jensen-Shannon (JS) divergence for all
complex environments. The fixed logarithmic type of reward function may be
difficult to solve all complex tasks, and the vanishing gradients problem
caused by the JS divergence will harm the adversarial learning process. In this
paper, we propose a new algorithm named Wasserstein Distance guided Adversarial
Imitation Learning (WDAIL) for promoting the performance of imitation learning
(IL). There are three improvements in our method: (a) introducing the
Wasserstein distance to obtain more appropriate measure in the adversarial
training process, (b) using proximal policy optimization (PPO) in the
reinforcement learning stage which is much simpler to implement and makes the
algorithm more efficient, and (c) exploring different reward function shapes to
suit different tasks for improving the performance. The experiment results show
that the learning procedure remains remarkably stable, and achieves significant
performance in the complex continuous control tasks of MuJoCo.Comment: M. Zhang and Y. Wang contribute equally to this wor
Learning Agile Skills via Adversarial Imitation of Rough Partial Demonstrations
Learning agile skills is one of the main challenges in robotics. To this end,
reinforcement learning approaches have achieved impressive results. These
methods require explicit task information in terms of a reward function or an
expert that can be queried in simulation to provide a target control output,
which limits their applicability. In this work, we propose a generative
adversarial method for inferring reward functions from partial and potentially
physically incompatible demonstrations for successful skill acquirement where
reference or expert demonstrations are not easily accessible. Moreover, we show
that by using a Wasserstein GAN formulation and transitions from demonstrations
with rough and partial information as input, we are able to extract policies
that are robust and capable of imitating demonstrated behaviors. Finally, the
obtained skills such as a backflip are tested on an agile quadruped robot
called Solo 8 and present faithful replication of hand-held human
demonstrations
Imitation Learning with Sinkhorn Distances
Imitation learning algorithms have been interpreted as variants of divergence
minimization problems. The ability to compare occupancy measures between
experts and learners is crucial in their effectiveness in learning from
demonstrations. In this paper, we present tractable solutions by formulating
imitation learning as minimization of the Sinkhorn distance between occupancy
measures. The formulation combines the valuable properties of optimal transport
metrics in comparing non-overlapping distributions with a cosine distance cost
defined in an adversarially learned feature space. This leads to a highly
discriminative critic network and optimal transport plan that subsequently
guide imitation learning. We evaluate the proposed approach using both the
reward metric and the Sinkhorn distance metric on a number of MuJoCo
experiments
Hierarchical Generative Adversarial Imitation Learning with Mid-level Input Generation for Autonomous Driving on Urban Environments
Deriving robust control policies for realistic urban navigation scenarios is
not a trivial task. In an end-to-end approach, these policies must map
high-dimensional images from the vehicle's cameras to low-level actions such as
steering and throttle. While pure Reinforcement Learning (RL) approaches are
based exclusively on rewards,Generative Adversarial Imitation Learning (GAIL)
agents learn from expert demonstrations while interacting with the environment,
which favors GAIL on tasks for which a reward signal is difficult to derive. In
this work, the hGAIL architecture was proposed to solve the autonomous
navigation of a vehicle in an end-to-end approach, mapping sensory perceptions
directly to low-level actions, while simultaneously learning mid-level input
representations of the agent's environment. The proposed hGAIL consists of an
hierarchical Adversarial Imitation Learning architecture composed of two main
modules: the GAN (Generative Adversarial Nets) which generates the Bird's-Eye
View (BEV) representation mainly from the images of three frontal cameras of
the vehicle, and the GAIL which learns to control the vehicle based mainly on
the BEV predictions from the GAN as input.Our experiments have shown that GAIL
exclusively from cameras (without BEV) fails to even learn the task, while
hGAIL, after training, was able to autonomously navigate successfully in all
intersections of the city
Improving Deep Reinforcement Learning Using Graph Convolution and Visual Domain Transfer
Recent developments in Deep Reinforcement Learning (DRL) have shown tremendous progress in robotics control, Atari games, board games such as Go, etc. However, model free DRL still has limited use cases due to its poor sampling efficiency and generalization on a variety of tasks. In this thesis, two particular drawbacks of DRL are investigated: 1) the poor generalization abilities of model free DRL. More specifically, how to generalize an agent\u27s policy to unseen environments and generalize to task performance on different data representations (e.g. image based or graph based) 2) The reality gap issue in DRL. That is, how to effectively transfer a policy learned in a simulator to the real world. This thesis makes several novel contributions to the field of DRL which are outlined sequentially in the following. Among these contributions is the generalized value iteration network (GVIN) algorithm, which is an end-to-end neural network planning module extending the work of Value Iteration Networks (VIN). GVIN emulates the value iteration algorithm by using a novel graph convolution operator, which enables GVIN to learn and plan on irregular spatial graphs. Additionally, this thesis proposes three novel, differentiable kernels as graph convolution operators and shows that the embedding-based kernel achieves the best performance. Furthermore, an improvement upon traditional -step -learning that stabilizes training for VIN and GVIN is demonstrated. Additionally, the equivalence between GVIN and graph neural networks is outlined and shown that GVIN can be further extended to address both control and inference problems. The final subject which falls under the graph domain that is studied in this thesis is graph embeddings. Specifically, this work studies a general graph embedding framework GEM-F that unifies most of the previous graph embedding algorithms. Based on the contributions made during the analysis of GEM-F, a novel algorithm called WarpMap which outperforms DeepWalk and node2vec in the unsupervised learning settings is proposed. The aforementioned reality gap in DRL prohibits a significant portion of research from reaching the real world setting. The latter part of this work studies and analyzes domain transfer techniques in an effort to bridge this gap. Typically, domain transfer in RL consists of representation transfer and policy transfer. In this work, the focus is on representation transfer for vision based applications. More specifically, aligning the feature representation from source domain to target domain in an unsupervised fashion. In this approach, a linear mapping function is considered to fuse modules that are trained in different domains. Proposed are two improved adversarial learning methods to enhance the training quality of the mapping function. Finally, the thesis demonstrates the effectiveness of domain alignment among different weather conditions in the CARLA autonomous driving simulator
Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizability
A trustworthy reinforcement learning algorithm should be competent in solving
challenging real-world problems, including {robustly} handling uncertainties,
satisfying {safety} constraints to avoid catastrophic failures, and
{generalizing} to unseen scenarios during deployments. This study aims to
overview these main perspectives of trustworthy reinforcement learning
considering its intrinsic vulnerabilities on robustness, safety, and
generalizability. In particular, we give rigorous formulations, categorize
corresponding methodologies, and discuss benchmarks for each perspective.
Moreover, we provide an outlook section to spur promising future directions
with a brief discussion on extrinsic vulnerabilities considering human
feedback. We hope this survey could bring together separate threads of studies
together in a unified framework and promote the trustworthiness of
reinforcement learning.Comment: 36 pages, 5 figure
On discovering and learning structure under limited supervision
Les formes, les surfaces, les événements et les objets (vivants et non vivants) constituent le monde. L'intelligence des agents naturels, tels que les humains, va au-delà de la simple reconnaissance de formes. Nous excellons à construire des représentations et à distiller des connaissances pour comprendre et déduire la structure du monde. Spécifiquement, le développement de telles capacités de raisonnement peut se produire même avec une supervision limitée.
D'autre part, malgré son développement phénoménal, les succès majeurs de l'apprentissage automatique, en particulier des modèles d'apprentissage profond, se situent principalement dans les tâches qui ont accès à de grands ensembles de données annotées. Dans cette thèse, nous proposons de nouvelles solutions pour aider à combler cette lacune en permettant aux modèles d'apprentissage automatique d'apprendre la structure et de permettre un raisonnement efficace en présence de tâches faiblement supervisés.
Le thème récurrent de la thèse tente de s'articuler autour de la question « Comment un système perceptif peut-il apprendre à organiser des informations sensorielles en connaissances utiles sous une supervision limitée ? » Et il aborde les thèmes de la géométrie, de la composition et des associations dans quatre articles distincts avec des applications à la vision par ordinateur (CV) et à l'apprentissage par renforcement (RL).
Notre première contribution ---Pix2Shape---présente une approche basée sur l'analyse par synthèse pour la perception. Pix2Shape exploite des modèles génératifs probabilistes pour apprendre des représentations 3D à partir d'images 2D uniques. Le formalisme qui en résulte nous offre une nouvelle façon de distiller l'information d'une scène ainsi qu'une représentation puissantes des images. Nous y parvenons en augmentant l'apprentissage profond non supervisé avec des biais inductifs basés sur la physique pour décomposer la structure causale des images en géométrie, orientation, pose, réflectance et éclairage.
Notre deuxième contribution ---MILe--- aborde les problèmes d'ambiguïté dans les ensembles de données à label unique tels que ImageNet. Il est souvent inapproprié de décrire une image avec un seul label lorsqu'il est composé de plus d'un objet proéminent. Nous montrons que l'intégration d'idées issues de la littérature linguistique cognitive et l'imposition de biais inductifs appropriés aident à distiller de multiples descriptions possibles à l'aide d'ensembles de données aussi faiblement étiquetés.
Ensuite, nous passons au paradigme d'apprentissage par renforcement, et considérons un agent interagissant avec son environnement sans signal de récompense. Notre troisième contribution ---HaC--- est une approche non supervisée basée sur la curiosité pour apprendre les associations entre les modalités visuelles et tactiles. Cela aide l'agent à explorer l'environnement de manière autonome et à utiliser davantage ses connaissances pour s'adapter aux tâches en aval. La supervision dense des récompenses n'est pas toujours disponible (ou n'est pas facile à concevoir), dans de tels cas, une exploration efficace est utile pour générer un comportement significatif de manière auto-supervisée.
Pour notre contribution finale, nous abordons l'information limitée contenue dans les représentations obtenues par des agents RL non supervisés. Ceci peut avoir un effet néfaste sur la performance des agents lorsque leur perception est basée sur des images de haute dimension. Notre approche a base de modèles combine l'exploration et la planification sans récompense pour affiner efficacement les modèles pré-formés non supervisés, obtenant des résultats comparables à un agent entraîné spécifiquement sur ces tâches. Il s'agit d'une étape vers la création d'agents capables de généraliser rapidement à plusieurs tâches en utilisant uniquement des images comme perception.Shapes, surfaces, events, and objects (living and non-living) constitute the world. The intelligence of natural agents, such as humans is beyond pattern recognition. We excel at building representations and distilling knowledge to understand and infer the structure of the world. Critically, the development of such reasoning capabilities can occur even with limited supervision.
On the other hand, despite its phenomenal development, the major successes of machine learning, in particular, deep learning models are primarily in tasks that have access to large annotated datasets. In this dissertation, we propose novel solutions to help address this gap by enabling machine learning models to learn the structure and enable effective reasoning in the presence of weakly supervised settings.
The recurring theme of the thesis tries to revolve around the question of "How can a perceptual system learn to organize sensory information into useful knowledge under limited supervision?" And it discusses the themes of geometry, compositions, and associations in four separate articles with applications to computer vision (CV) and reinforcement learning (RL).
Our first contribution ---Pix2Shape---presents an analysis-by-synthesis based approach(also referred to as inverse graphics) for perception. Pix2Shape leverages probabilistic generative models to learn 3D-aware representations from single 2D images. The resulting formalism allows us to perform a novel view synthesis of a scene and produce powerful representations of images. We achieve this by augmenting unsupervised learning with physically based inductive biases to decompose a scene structure into geometry, pose, reflectance and lighting.
Our Second contribution ---MILe--- addresses the ambiguity issues in single-labeled datasets such as ImageNet. It is often inappropriate to describe an image with a single label when it is composed of more than one prominent object. We show that integrating ideas from Cognitive linguistic literature and imposing appropriate inductive biases helps in distilling multiple possible descriptions using such weakly labeled datasets.
Next, moving into the RL setting, we consider an agent interacting with its environment without a reward signal. Our third Contribution ---HaC--- is a curiosity based unsupervised approach to learning associations between visual and tactile modalities. This aids the agent to explore the environment in an analogous self-guided fashion and further use this knowledge to adapt to downstream tasks.
In the absence of reward supervision, intrinsic movitivation is useful to generate meaningful behavior in a self-supervised manner.
In our final contribution, we address the representation learning bottleneck in unsupervised RL agents that has detrimental effect on the performance on high-dimensional pixel based inputs. Our model-based approach combines reward-free exploration and planning to efficiently fine-tune unsupervised pre-trained models, achieving comparable results to task-specific baselines. This is a step towards building agents that can generalize quickly on more than a single task using image inputs alone
Reward is enough for convex MDPs
Maximising a cumulative reward function that is Markov and stationary, i.e.,
defined over state-action pairs and independent of time, is sufficient to
capture many kinds of goals in a Markov decision process (MDP). However, not
all goals can be captured in this manner. In this paper we study convex MDPs in
which goals are expressed as convex functions of the stationary distribution
and show that they cannot be formulated using stationary reward functions.
Convex MDPs generalize the standard reinforcement learning (RL) problem
formulation to a larger framework that includes many supervised and
unsupervised RL problems, such as apprenticeship learning, constrained MDPs,
and so-called `pure exploration'. Our approach is to reformulate the convex MDP
problem as a min-max game involving policy and cost (negative reward)
`players', using Fenchel duality. We propose a meta-algorithm for solving this
problem and show that it unifies many existing algorithms in the literature
- …