1,120 research outputs found
Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey
Wireless sensor networks (WSNs) consist of autonomous and resource-limited
devices. The devices cooperate to monitor one or more physical phenomena within
an area of interest. WSNs operate as stochastic systems because of randomness
in the monitored environments. For long service time and low maintenance cost,
WSNs require adaptive and robust methods to address data exchange, topology
formulation, resource and power optimization, sensing coverage and object
detection, and security challenges. In these problems, sensor nodes are to make
optimized decisions from a set of accessible strategies to achieve design
goals. This survey reviews numerous applications of the Markov decision process
(MDP) framework, a powerful decision-making tool to develop adaptive algorithms
and protocols for WSNs. Furthermore, various solution methods are discussed and
compared to serve as a guide for using MDPs in WSNs
Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks
Future wireless networks have a substantial potential in terms of supporting
a broad range of complex compelling applications both in military and civilian
fields, where the users are able to enjoy high-rate, low-latency, low-cost and
reliable information services. Achieving this ambitious goal requires new radio
techniques for adaptive learning and intelligent decision making because of the
complex heterogeneous nature of the network structures and wireless services.
Machine learning (ML) algorithms have great success in supporting big data
analytics, efficient parameter estimation and interactive decision making.
Hence, in this article, we review the thirty-year history of ML by elaborating
on supervised learning, unsupervised learning, reinforcement learning and deep
learning. Furthermore, we investigate their employment in the compelling
applications of wireless networks, including heterogeneous networks (HetNets),
cognitive radios (CR), Internet of things (IoT), machine to machine networks
(M2M), and so on. This article aims for assisting the readers in clarifying the
motivation and methodology of the various ML algorithms, so as to invoke them
for hitherto unexplored services as well as scenarios of future wireless
networks.Comment: 46 pages, 22 fig
Energy Sharing for Multiple Sensor Nodes with Finite Buffers
We consider the problem of finding optimal energy sharing policies that
maximize the network performance of a system comprising of multiple sensor
nodes and a single energy harvesting (EH) source. Sensor nodes periodically
sense the random field and generate data, which is stored in the corresponding
data queues. The EH source harnesses energy from ambient energy sources and the
generated energy is stored in an energy buffer. Sensor nodes receive energy for
data transmission from the EH source. The EH source has to efficiently share
the stored energy among the nodes in order to minimize the long-run average
delay in data transmission. We formulate the problem of energy sharing between
the nodes in the framework of average cost infinite-horizon Markov decision
processes (MDPs). We develop efficient energy sharing algorithms, namely
Q-learning algorithm with exploration mechanisms based on the -greedy
method as well as upper confidence bound (UCB). We extend these algorithms by
incorporating state and action space aggregation to tackle state-action space
explosion in the MDP. We also develop a cross entropy based method that
incorporates policy parameterization in order to find near optimal energy
sharing policies. Through simulations, we show that our algorithms yield energy
sharing policies that outperform the heuristic greedy method.Comment: 38 pages, 10 figure
Stuck in Traffic (SiT) Attacks: A Framework for Identifying Stealthy Attacks that Cause Traffic Congestion
Recent advances in wireless technologies have enabled many new applications
in Intelligent Transportation Systems (ITS) such as collision avoidance,
cooperative driving, congestion avoidance, and traffic optimization. Due to the
vulnerable nature of wireless communication against interference and
intentional jamming, ITS face new challenges to ensure the reliability and the
safety of the overall system. In this paper, we expose a class of stealthy
attacks -- Stuck in Traffic (SiT) attacks -- that aim to cause congestion by
exploiting how drivers make decisions based on smart traffic signs. An attacker
mounting a SiT attack solves a Markov Decision Process problem to find
optimal/suboptimal attack policies in which he/she interferes with a
well-chosen subset of signals that are based on the state of the system. We
apply Approximate Policy Iteration (API) algorithms to derive potent attack
policies. We evaluate their performance on a number of systems and compare them
to other attack policies including random, myopic and DoS attack policies. The
generated policies, albeit suboptimal, are shown to significantly outperform
other attack policies as they maximize the expected cumulative reward from the
standpoint of the attacker
A Cross-layer Perspective on Energy Harvesting Aided Green Communications over Fading Channels
We consider the power allocation of the physical layer and the buffer delay
of the upper application layer in energy harvesting green networks. The total
power required for reliable transmission includes the transmission power and
the circuit power. The harvested power (which is stored in a battery) and the
grid power constitute the power resource. The uncertainty of data generated
from the upper layer, the intermittence of the harvested energy, and the
variation of the fading channel are taken into account and described as
independent Markov processes. In each transmission, the transmitter decides the
transmission rate as well as the allocated power from the battery, and the rest
of the required power will be supplied by the power grid. The objective is to
find an allocation sequence of transmission rate and battery power to minimize
the long-term average buffer delay under the average grid power constraint. A
stochastic optimization problem is formulated accordingly to find such
transmission rate and battery power sequence. Furthermore, the optimization
problem is reformulated as a constrained MDP problem whose policy is a
two-dimensional vector with the transmission rate and the power allocation of
the battery as its elements. We prove that the optimal policy of the
constrained MDP can be obtained by solving the unconstrained MDP. Then we focus
on the analysis of the unconstrained average-cost MDP. The structural
properties of the average optimal policy are derived. Moreover, we discuss the
relations between elements of the two-dimensional policy. Next, based on the
theoretical analysis, the algorithm to find the constrained optimal policy is
presented for the finite state space scenario. In addition, heuristic policies
with low-complexity are given for the general state space. Finally, simulations
are performed under these policies to demonstrate the effectiveness
A Survey on Delay-Aware Resource Control for Wireless Systems --- Large Deviation Theory, Stochastic Lyapunov Drift and Distributed Stochastic Learning
In this tutorial paper, a comprehensive survey is given on several major
systematic approaches in dealing with delay-aware control problems, namely the
equivalent rate constraint approach, the Lyapunov stability drift approach and
the approximate Markov Decision Process (MDP) approach using stochastic
learning. These approaches essentially embrace most of the existing literature
regarding delay-aware resource control in wireless systems. They have their
relative pros and cons in terms of performance, complexity and implementation
issues. For each of the approaches, the problem setup, the general solution and
the design methodology are discussed. Applications of these approaches to
delay-aware resource allocation are illustrated with examples in single-hop
wireless networks. Furthermore, recent results regarding delay-aware multi-hop
routing designs in general multi-hop networks are elaborated. Finally, the
delay performance of the various approaches are compared through simulations
using an example of the uplink OFDMA systems.Comment: 58 pages, 8 figures; IEEE Transactions on Information Theory, 201
- …