4 research outputs found
Performance Dynamics and Termination Errors in Reinforcement Learning: A Unifying Perspective
In reinforcement learning, a decision needs to be made at some point as to
whether it is worthwhile to carry on with the learning process or to terminate
it. In many such situations, stochastic elements are often present which govern
the occurrence of rewards, with the sequential occurrences of positive rewards
randomly interleaved with negative rewards. For most practical learners, the
learning is considered useful if the number of positive rewards always exceeds
the negative ones. A situation that often calls for learning termination is
when the number of negative rewards exceeds the number of positive rewards.
However, while this seems reasonable, the error of premature termination,
whereby termination is enacted along with the conclusion of learning failure
despite the positive rewards eventually far outnumber the negative ones, can be
significant. In this paper, using combinatorial analysis we study the error
probability in wrongly terminating a reinforcement learning activity which
undermines the effectiveness of an optimal policy, and we show that the
resultant error can be quite high. Whilst we demonstrate mathematically that
such errors can never be eliminated, we propose some practical mechanisms that
can effectively reduce such errors. Simulation experiments have been carried
out, the results of which are in close agreement with our theoretical findings.Comment: Short Paper in AIKE 201
Stochastic Reinforcement Learning
In reinforcement learning episodes, the rewards and punishments are often
non-deterministic, and there are invariably stochastic elements governing the
underlying situation. Such stochastic elements are often numerous and cannot be
known in advance, and they have a tendency to obscure the underlying rewards
and punishments patterns. Indeed, if stochastic elements were absent, the same
outcome would occur every time and the learning problems involved could be
greatly simplified. In addition, in most practical situations, the cost of an
observation to receive either a reward or punishment can be significant, and
one would wish to arrive at the correct learning conclusion by incurring
minimum cost. In this paper, we present a stochastic approach to reinforcement
learning which explicitly models the variability present in the learning
environment and the cost of observation. Criteria and rules for learning
success are quantitatively analyzed, and probabilities of exceeding the
observation cost bounds are also obtained.Comment: AIKE 201
Statistical and Computational Trade-offs in Variational Inference: A Case Study in Inferential Model Selection
Variational inference has recently emerged as a popular alternative to the
classical Markov chain Monte Carlo (MCMC) in large-scale Bayesian inference.
The core idea of variational inference is to trade statistical accuracy for
computational efficiency. It aims to approximate the posterior, reducing
computation costs but potentially compromising its statistical accuracy. In
this work, we study this statistical and computational trade-off in variational
inference via a case study in inferential model selection. Focusing on Gaussian
inferential models (a.k.a. variational approximating families) with diagonal
plus low-rank precision matrices, we initiate a theoretical study of the
trade-offs in two aspects, Bayesian posterior inference error and frequentist
uncertainty quantification error. From the Bayesian posterior inference
perspective, we characterize the error of the variational posterior relative to
the exact posterior. We prove that, given a fixed computation budget, a
lower-rank inferential model produces variational posteriors with a higher
statistical approximation error, but a lower computational error; it reduces
variances in stochastic optimization and, in turn, accelerates convergence.
From the frequentist uncertainty quantification perspective, we consider the
precision matrix of the variational posterior as an uncertainty estimate. We
find that, relative to the true asymptotic precision, the variational
approximation suffers from an additional statistical error originating from the
sampling uncertainty of the data. Moreover, this statistical error becomes the
dominant factor as the computation budget increases. As a consequence, for
small datasets, the inferential model need not be full-rank to achieve optimal
estimation error. We finally demonstrate these statistical and computational
trade-offs inference across empirical studies, corroborating the theoretical
findings.Comment: 56 pages, 8 figure