18,384 research outputs found
Performance Guarantees for Homomorphisms Beyond Markov Decision Processes
Most real-world problems have huge state and/or action spaces. Therefore, a
naive application of existing tabular solution methods is not tractable on such
problems. Nonetheless, these solution methods are quite useful if an agent has
access to a relatively small state-action space homomorphism of the true
environment and near-optimal performance is guaranteed by the map. A plethora
of research is focused on the case when the homomorphism is a Markovian
representation of the underlying process. However, we show that near-optimal
performance is sometimes guaranteed even if the homomorphism is non-Markovian.
Moreover, we can aggregate significantly more states by lifting the Markovian
requirement without compromising on performance. In this work, we expand
Extreme State Aggregation (ESA) framework to joint state-action aggregations.
We also lift the policy uniformity condition for aggregation in ESA that allows
even coarser modeling of the true environment
An Analysis of the Value of Information when Exploring Stochastic, Discrete Multi-Armed Bandits
In this paper, we propose an information-theoretic exploration strategy for
stochastic, discrete multi-armed bandits that achieves optimal regret. Our
strategy is based on the value of information criterion. This criterion
measures the trade-off between policy information and obtainable rewards. High
amounts of policy information are associated with exploration-dominant searches
of the space and yield high rewards. Low amounts of policy information favor
the exploitation of existing knowledge. Information, in this criterion, is
quantified by a parameter that can be varied during search. We demonstrate that
a simulated-annealing-like update of this parameter, with a sufficiently fast
cooling schedule, leads to an optimal regret that is logarithmic with respect
to the number of episodes.Comment: Entrop
- …