19 research outputs found
The Gossiping Insert-Eliminate Algorithm for Multi-Agent Bandits
We consider a decentralized multi-agent Multi Armed Bandit (MAB) setup
consisting of agents, solving the same MAB instance to minimize individual
cumulative regret. In our model, agents collaborate by exchanging messages
through pairwise gossip style communications on an arbitrary connected graph.
We develop two novel algorithms, where each agent only plays from a subset of
all the arms. Agents use the communication medium to recommend only arm-IDs
(not samples), and thus update the set of arms from which they play. We
establish that, if agents communicate times through any
connected pairwise gossip mechanism, then every agent's regret is a factor of
order smaller compared to the case of no collaborations. Furthermore, we
show that the communication constraints only have a second order effect on the
regret of our algorithm. We then analyze this second order term of the regret
to derive bounds on the regret-communication tradeoffs. Finally, we empirically
evaluate our algorithm and conclude that the insights are fundamental and not
artifacts of our bounds. We also show a lower bound which gives that the regret
scaling obtained by our algorithm cannot be improved even in the absence of any
communication constraints. Our results thus demonstrate that even a minimal
level of collaboration among agents greatly reduces regret for all agents.Comment: To Appear in AISTATS 2020. The first two authors contributed equall
Multi-Agent Low-Dimensional Linear Bandits
We study a multi-agent stochastic linear bandit with side information,
parameterized by an unknown vector . The side
information consists of a finite collection of low-dimensional subspaces, one
of which contains . In our setting, agents can collaborate to reduce
regret by sending recommendations across a communication graph connecting them.
We present a novel decentralized algorithm, where agents communicate subspace
indices with each other, and each agent plays a projected variant of LinUCB on
the corresponding (low-dimensional) subspace. Through a combination of
collaborative best subspace identification, and per-agent learning of an
unknown vector in the corresponding low-dimensional subspace, we show that the
per-agent regret is much smaller than the case when agents do not communicate.
By collaborating to identify the subspace containing , we show that
each agent effectively solves an easier instance of the linear bandit (compared
to the case of no collaboration), thus leading to the reduced per-agent regret.
We finally complement these results through simulations
Communication-Efficient Collaborative Regret Minimization in Multi-Armed Bandits
In this paper, we study the collaborative learning model, which concerns the
tradeoff between parallelism and communication overhead in multi-agent
multi-armed bandits. For regret minimization in multi-armed bandits, we present
the first set of tradeoffs between the number of rounds of communication among
the agents and the regret of the collaborative learning process.Comment: 13 pages, 1 figur
Cooperative Thresholded Lasso for Sparse Linear Bandit
We present a novel approach to address the multi-agent sparse contextual
linear bandit problem, in which the feature vectors have a high dimension
whereas the reward function depends on only a limited set of features -
precisely . Furthermore, the learning follows under
information-sharing constraints. The proposed method employs Lasso regression
for dimension reduction, allowing each agent to independently estimate an
approximate set of main dimensions and share that information with others
depending on the network's structure. The information is then aggregated
through a specific process and shared with all agents. Each agent then resolves
the problem with ridge regression focusing solely on the extracted dimensions.
We represent algorithms for both a star-shaped network and a peer-to-peer
network. The approaches effectively reduce communication costs while ensuring
minimal cumulative regret per agent. Theoretically, we show that our proposed
methods have a regret bound of order
with high probability, where is the time horizon. To our best knowledge, it
is the first algorithm that tackles row-wise distributed data in sparse linear
bandits, achieving comparable performance compared to the state-of-the-art
single and multi-agent methods. Besides, it is widely applicable to
high-dimensional multi-agent problems where efficient feature extraction is
critical for minimizing regret. To validate the effectiveness of our approach,
we present experimental results on both synthetic and real-world datasets
On-Demand Communication for Asynchronous Multi-Agent Bandits
This paper studies a cooperative multi-agent multi-armed stochastic bandit
problem where agents operate asynchronously -- agent pull times and rates are
unknown, irregular, and heterogeneous -- and face the same instance of a
K-armed bandit problem. Agents can share reward information to speed up the
learning process at additional communication costs. We propose ODC, an
on-demand communication protocol that tailors the communication of each pair of
agents based on their empirical pull times. ODC is efficient when the pull
times of agents are highly heterogeneous, and its communication complexity
depends on the empirical pull times of agents. ODC is a generic protocol that
can be integrated into most cooperative bandit algorithms without degrading
their performance. We then incorporate ODC into the natural extensions of UCB
and AAE algorithms and propose two communication-efficient cooperative
algorithms. Our analysis shows that both algorithms are near-optimal in regret.Comment: Accepted by AISTATS 202
Collaborative Multi-Agent Heterogeneous Multi-Armed Bandits
The study of collaborative multi-agent bandits has attracted significant
attention recently. In light of this, we initiate the study of a new
collaborative setting, consisting of agents such that each agent is
learning one of stochastic multi-armed bandits to minimize their group
cumulative regret. We develop decentralized algorithms which facilitate
collaboration between the agents under two scenarios. We characterize the
performance of these algorithms by deriving the per agent cumulative regret and
group regret upper bounds. We also prove lower bounds for the group regret in
this setting, which demonstrates the near-optimal behavior of the proposed
algorithms.Comment: To appear in the proceedings of ICML 202
Tractable Optimality in Episodic Latent MABs
We consider a multi-armed bandit problem with latent contexts, where an
agent interacts with the environment for an episode of time steps.
Depending on the length of the episode, the learner may not be able to estimate
accurately the latent context. The resulting partial observation of the
environment makes the learning task significantly more challenging. Without any
additional structural assumptions, existing techniques to tackle partially
observed settings imply the decision maker can learn a near-optimal policy with
episodes, but do not promise more. In this work, we show that learning
with {\em polynomial} samples in is possible. We achieve this by using
techniques from experiment design. Then, through a method-of-moments approach,
we design a procedure that provably learns a near-optimal policy with
interactions. In
practice, we show that we can formulate the moment-matching via maximum
likelihood estimation. In our experiments, this significantly outperforms the
worst-case guarantees, as well as existing practical methods.Comment: NeurIPS 202