25 research outputs found
Optimal Model Selection in Contextual Bandits with Many Classes via Offline Oracles
We study the problem of model selection for contextual bandits, in which the
algorithm must balance the bias-variance trade-off for model estimation while
also balancing the exploration-exploitation trade-off. In this paper, we
propose the first reduction of model selection in contextual bandits to offline
model selection oracles, allowing for flexible general purpose algorithms with
computational requirements no worse than those for model selection for
regression. Our main result is a new model selection guarantee for stochastic
contextual bandits. When one of the classes in our set is realizable, up to a
logarithmic dependency on the number of classes, our algorithm attains optimal
realizability-based regret bounds for that class under one of two conditions:
if the time-horizon is large enough, or if an assumption that helps with
detecting misspecification holds. Hence our algorithm adapts to the complexity
of this unknown class. Even when this realizable class is known, we prove
improved regret guarantees in early rounds by relying on simpler model classes
for those rounds and hence further establish the importance of model selection
in contextual bandits
Model Selection for Generic Contextual Bandits
We consider the problem of model selection for the general stochastic
contextual bandits under the realizability assumption. We propose a successive
refinement based algorithm called Adaptive Contextual Bandit ({\ttfamily ACB}),
that works in phases and successively eliminates model classes that are too
simple to fit the given instance. We prove that this algorithm is adaptive,
i.e., the regret rate order-wise matches that of {\ttfamily FALCON}, the
state-of-art contextual bandit algorithm of Levi et. al '20, that needs
knowledge of the true model class. The price of not knowing the correct model
class is only an additive term contributing to the second order term in the
regret bound. This cost possess the intuitive property that it becomes smaller
as the model class becomes easier to identify, and vice-versa. We then show
that a much simpler explore-then-commit (ETC) style algorithm also obtains a
regret rate of matching that of {\ttfamily FALCON}, despite not knowing the
true model class. However, the cost of model selection is higher in ETC as
opposed to in {\ttfamily ACB}, as expected. Furthermore, {\ttfamily ACB}
applied to the linear bandit setting with unknown sparsity, order-wise recovers
the model selection guarantees previously established by algorithms tailored to
the linear setting.Comment: 40 pages, 5 figures. arXiv admin note: text overlap with
arXiv:2006.0261
Efficient and Robust Algorithms for Adversarial Linear Contextual Bandits
We consider an adversarial variant of the classic -armed linear contextual
bandit problem where the sequence of loss functions associated with each arm
are allowed to change without restriction over time. Under the assumption that
the -dimensional contexts are generated i.i.d.~at random from a known
distributions, we develop computationally efficient algorithms based on the
classic Exp3 algorithm. Our first algorithm, RealLinExp3, is shown to achieve a
regret guarantee of over rounds, which matches
the best available bound for this problem. Our second algorithm, RobustLinExp3,
is shown to be robust to misspecification, in that it achieves a regret bound
of if the true
reward function is linear up to an additive nonlinear error uniformly bounded
in absolute value by . To our knowledge, our performance
guarantees constitute the very first results on this problem setting