19 research outputs found
Sampling Through the Lens of Sequential Decision Making
Sampling is ubiquitous in machine learning methodologies. Due to the growth
of large datasets and model complexity, we want to learn and adapt the sampling
process while training a representation. Towards achieving this grand goal, a
variety of sampling techniques have been proposed. However, most of them either
use a fixed sampling scheme or adjust the sampling scheme based on simple
heuristics. They cannot choose the best sample for model training in different
stages. Inspired by "Think, Fast and Slow" (System 1 and System 2) in cognitive
science, we propose a reward-guided sampling strategy called Adaptive Sample
with Reward (ASR) to tackle this challenge. To the best of our knowledge, this
is the first work utilizing reinforcement learning (RL) to address the sampling
problem in representation learning. Our approach optimally adjusts the sampling
process to achieve optimal performance. We explore geographical relationships
among samples by distance-based sampling to maximize overall cumulative reward.
We apply ASR to the long-standing sampling problems in similarity-based loss
functions. Empirical results in information retrieval and clustering
demonstrate ASR's superb performance across different datasets. We also discuss
an engrossing phenomenon which we name as "ASR gravity well" in experiments
Momentum-Based Policy Gradient with Second-Order Information
Variance-reduced gradient estimators for policy gradient methods have been
one of the main focus of research in the reinforcement learning in recent years
as they allow acceleration of the estimation process. We propose a
variance-reduced policy-gradient method, called SHARP, which incorporates
second-order information into stochastic gradient descent (SGD) using momentum
with a time-varying learning rate. SHARP algorithm is parameter-free, achieving
-approximate first-order stationary point with
number of trajectories, while using a batch size of at each iteration.
Unlike most previous work, our proposed algorithm does not require importance
sampling which can compromise the advantage of variance reduction process.
Moreover, the variance of estimation error decays with the fast rate of
where is the number of iterations. Our extensive
experimental evaluations show the effectiveness of the proposed algorithm on
various control tasks and its advantage over the state of the art in practice
Improved Sample Complexity Analysis of Natural Policy Gradient Algorithm with General Parameterization for Infinite Horizon Discounted Reward Markov Decision Processes
We consider the problem of designing sample efficient learning algorithms for
infinite horizon discounted reward Markov Decision Process. Specifically, we
propose the Accelerated Natural Policy Gradient (ANPG) algorithm that utilizes
an accelerated stochastic gradient descent process to obtain the natural policy
gradient. ANPG achieves sample complexity and
iteration complexity with general parameterization
where defines the optimality error. This improves the
state-of-the-art sample complexity by a factor. ANPG
is a first-order algorithm and unlike some existing literature, does not
require the unverifiable assumption that the variance of importance sampling
(IS) weights is upper bounded. In the class of Hessian-free and IS-free
algorithms, ANPG beats the best-known sample complexity by a factor of
and simultaneously matches their
state-of-the-art iteration complexity
Diagnostic Evaluation of Policy-Gradient-Based Ranking
Learning-to-rank has been intensively studied and has shown significantly increasing values in a wide range of domains, such as web search, recommender systems, dialogue systems, machine translation, and even computational biology, to name a few. In light of recent advances in neural networks, there has been a strong and continuing interest in exploring how to deploy popular techniques, such as reinforcement learning and adversarial learning, to solve ranking problems. However, armed with the aforesaid popular techniques, most studies tend to show how effective a new method is. A comprehensive comparison between techniques and an in-depth analysis of their deficiencies are somehow overlooked. This paper is motivated by the observation that recent ranking methods based on either reinforcement learning or adversarial learning boil down to policy-gradient-based optimization. Based on the widely used benchmark collections with complete information (where relevance labels are known for all items), such as MSLRWEB30K and Yahoo-Set1, we thoroughly investigate the extent to which policy-gradient-based ranking methods are effective. On one hand, we analytically identify the pitfalls of policy-gradient-based ranking. On the other hand, we experimentally compare a wide range of representative methods. The experimental results echo our analysis and show that policy-gradient-based ranking methods are, by a large margin, inferior to many conventional ranking methods. Regardless of whether we use reinforcement learning or adversarial learning, the failures are largely attributable to the gradient estimation based on sampled rankings, which significantly diverge from ideal rankings. In particular, the larger the number of documents per query and the more fine-grained the ground-truth labels, the greater the impact policy-gradient-based ranking suffers. Careful examination of this weakness is highly recommended for developing enhanced methods based on policy gradient
A Cubic-regularized Policy Newton Algorithm for Reinforcement Learning
We consider the problem of control in the setting of reinforcement learning
(RL), where model information is not available. Policy gradient algorithms are
a popular solution approach for this problem and are usually shown to converge
to a stationary point of the value function. In this paper, we propose two
policy Newton algorithms that incorporate cubic regularization. Both algorithms
employ the likelihood ratio method to form estimates of the gradient and
Hessian of the value function using sample trajectories. The first algorithm
requires an exact solution of the cubic regularized problem in each iteration,
while the second algorithm employs an efficient gradient descent-based
approximation to the cubic regularized problem. We establish convergence of our
proposed algorithms to a second-order stationary point (SOSP) of the value
function, which results in the avoidance of traps in the form of saddle points.
In particular, the sample complexity of our algorithms to find an
-SOSP is , which is an improvement over the
state-of-the-art sample complexity of