772 research outputs found
Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey
Wireless sensor networks (WSNs) consist of autonomous and resource-limited
devices. The devices cooperate to monitor one or more physical phenomena within
an area of interest. WSNs operate as stochastic systems because of randomness
in the monitored environments. For long service time and low maintenance cost,
WSNs require adaptive and robust methods to address data exchange, topology
formulation, resource and power optimization, sensing coverage and object
detection, and security challenges. In these problems, sensor nodes are to make
optimized decisions from a set of accessible strategies to achieve design
goals. This survey reviews numerous applications of the Markov decision process
(MDP) framework, a powerful decision-making tool to develop adaptive algorithms
and protocols for WSNs. Furthermore, various solution methods are discussed and
compared to serve as a guide for using MDPs in WSNs
Pilot Beam Sequence Design for Channel Estimation in Millimeter-Wave MIMO Systems: A POMDP Framework
In this paper, adaptive pilot beam sequence design for channel estimation in
large millimeter-wave (mmWave) MIMO systems is considered. By exploiting the
sparsity of mmWave MIMO channels with the virtual channel representation and
imposing a Markovian random walk assumption on the physical movement of the
line-of-sight (LOS) and reflection clusters, it is shown that the sparse
channel estimation problem in large mmWave MIMO systems reduces to a sequential
detection problem that finds the locations and values of the non-zero-valued
bins in a two-dimensional rectangular grid, and the optimal adaptive pilot
design problem can be cast into the framework of a partially observable Markov
decision process (POMDP). Under the POMDP framework, an optimal adaptive pilot
beam sequence design method is obtained to maximize the accumulated
transmission data rate for a given period of time. Numerical results are
provided to validate our pilot signal design method and they show that the
proposed method yields good performance.Comment: 6 pages, 6 figures, submitted to IEEE ICC 201
Monte Carlo Bayesian Reinforcement Learning
Bayesian reinforcement learning (BRL) encodes prior knowledge of the world in
a model and represents uncertainty in model parameters by maintaining a
probability distribution over them. This paper presents Monte Carlo BRL
(MC-BRL), a simple and general approach to BRL. MC-BRL samples a priori a
finite set of hypotheses for the model parameter values and forms a discrete
partially observable Markov decision process (POMDP) whose state space is a
cross product of the state space for the reinforcement learning task and the
sampled model parameter space. The POMDP does not require conjugate
distributions for belief representation, as earlier works do, and can be solved
relatively easily with point-based approximation algorithms. MC-BRL naturally
handles both fully and partially observable worlds. Theoretical and
experimental results show that the discrete POMDP approximates the underlying
BRL task well with guaranteed performance.Comment: Appears in Proceedings of the 29th International Conference on
Machine Learning (ICML 2012
- …