17,197 research outputs found
Deep Multi-User Reinforcement Learning for Distributed Dynamic Spectrum Access
We consider the problem of dynamic spectrum access for network utility
maximization in multichannel wireless networks. The shared bandwidth is divided
into K orthogonal channels. In the beginning of each time slot, each user
selects a channel and transmits a packet with a certain transmission
probability. After each time slot, each user that has transmitted a packet
receives a local observation indicating whether its packet was successfully
delivered or not (i.e., ACK signal). The objective is a multi-user strategy for
accessing the spectrum that maximizes a certain network utility in a
distributed manner without online coordination or message exchanges between
users. Obtaining an optimal solution for the spectrum access problem is
computationally expensive in general due to the large state space and partial
observability of the states. To tackle this problem, we develop a novel
distributed dynamic spectrum access algorithm based on deep multi-user
reinforcement leaning. Specifically, at each time slot, each user maps its
current state to spectrum access actions based on a trained deep-Q network used
to maximize the objective function. Game theoretic analysis of the system
dynamics is developed for establishing design principles for the implementation
of the algorithm. Experimental results demonstrate strong performance of the
algorithm.Comment: This work has been accepted for publication in the IEEE Transactions
on Wireless Communication
Distributed Cooperative Spectrum Sharing in UAV Networks Using Multi-Agent Reinforcement Learning
In this paper, we develop a distributed mechanism for spectrum sharing among
a network of unmanned aerial vehicles (UAV) and licensed terrestrial networks.
This method can provide a practical solution for situations where the UAV
network may need external spectrum when dealing with congested spectrum or need
to change its operating frequency due to security threats. Here we study a
scenario where the UAV network performs a remote sensing mission. In this
model, the UAVs are categorized into two clusters of relaying and sensing UAVs.
The relay UAVs provide a relaying service for a licensed network to obtain
spectrum access for the rest of UAVs that perform the sensing task. We develop
a distributed mechanism in which the UAVs locally decide whether they need to
participate in relaying or sensing considering the fact that communications
among UAVs may not be feasible or reliable. The UAVs learn the optimal task
allocation using a distributed reinforcement learning algorithm. Convergence of
the algorithm is discussed and simulation results are presented for different
scenarios to verify the convergence.Comment: 16 Pages, 8 Figure
Dynamic Spectrum Access in Time-varying Environment: Distributed Learning Beyond Expectation Optimization
This article investigates the problem of dynamic spectrum access for
canonical wireless networks, in which the channel states are time-varying. In
the most existing work, the commonly used optimization objective is to maximize
the expectation of a certain metric (e.g., throughput or achievable rate).
However, it is realized that expectation alone is not enough since some
applications are sensitive to fluctuations. Effective capacity is a promising
metric for time-varying service process since it characterizes the packet delay
violating probability (regarded as an important statistical QoS index), by
taking into account not only the expectation but also other high-order
statistic. Therefore, we formulate the interactions among the users in the
time-varying environment as a non-cooperative game, in which the utility
function is defined as the achieved effective capacity. We prove that it is an
ordinal potential game which has at least one pure strategy Nash equilibrium.
Based on an approximated utility function, we propose a multi-agent learning
algorithm which is proved to achieve stable solutions with dynamic and
incomplete information constraints. The convergence of the proposed learning
algorithm is verified by simulation results. Also, it is shown that the
proposed multi-agent learning algorithm achieves satisfactory performance.Comment: 13 pages, 10 figures, accepted for publication in IEEE Transactions
on Communication
A Solution for Dynamic Spectrum Management in Mission-Critical UAV Networks
In this paper, we study the problem of spectrum scarcity in a network of
unmanned aerial vehicles (UAVs) during mission-critical applications such as
disaster monitoring and public safety missions, where the pre-allocated
spectrum is not sufficient to offer a high data transmission rate for real-time
video-streaming. In such scenarios, the UAV network can lease part of the
spectrum of a terrestrial licensed network in exchange for providing relaying
service. In order to optimize the performance of the UAV network and prolong
its lifetime, some of the UAVs will function as a relay for the primary network
while the rest of the UAVs carry out their sensing tasks. Here, we propose a
team reinforcement learning algorithm performed by the UAV's controller unit to
determine the optimum allocation of sensing and relaying tasks among the UAVs
as well as their relocation strategy at each time. We analyze the convergence
of our algorithm and present simulation results to evaluate the system
throughput in different scenarios.Comment: 10 Pages, 5 Figures, 2 Table
From 4G to 5G: Self-organized Network Management meets Machine Learning
In this paper, we provide an analysis of self-organized network management,
with an end-to-end perspective of the network. Self-organization as applied to
cellular networks is usually referred to Self-organizing Networks (SONs), and
it is a key driver for improving Operations, Administration, and Maintenance
(OAM) activities. SON aims at reducing the cost of installation and management
of 4G and future 5G networks, by simplifying operational tasks through the
capability to configure, optimize and heal itself. To satisfy 5G network
management requirements, this autonomous management vision has to be extended
to the end to end network. In literature and also in some instances of products
available in the market, Machine Learning (ML) has been identified as the key
tool to implement autonomous adaptability and take advantage of experience when
making decisions. In this paper, we survey how network management can
significantly benefit from ML solutions. We review and provide the basic
concepts and taxonomy for SON, network management and ML. We analyse the
available state of the art in the literature, standardization, and in the
market. We pay special attention to 3rd Generation Partnership Project (3GPP)
evolution in the area of network management and to the data that can be
extracted from 3GPP networks, in order to gain knowledge and experience in how
the network is working, and improve network performance in a proactive way.
Finally, we go through the main challenges associated with this line of
research, in both 4G and in what 5G is getting designed, while identifying new
directions for research.Comment: 23 pages, 3 figures, Surve
Power Allocation for Cognitive Wireless Mesh Networks by Applying Multi-agent Q-learning Approach
As the scarce spectrum resource is becoming over-crowded, cognitive radios
(CRs) indicate great flexibility to improve the spectrum efficiency by
opportunistically accessing the authorized frequency bands. One of the critical
challenges for operating such radios in a network is how to efficiently
allocate transmission powers and frequency resource among the secondary users
(SUs) while satisfying the quality-of-service (QoS) constraints of the primary
users (PUs). In this paper, we focus on the non-cooperative power allocation
problem in cognitive wireless mesh networks (CogMesh) formed by a number of
clusters with the consideration of energy efficiency. Due to the SUs' selfish
and spontaneous properties, the problem is modeled as a stochastic learning
process. We first extend the single-agent Q-learning to a multi-user context,
and then propose a conjecture based multi-agent Qlearning algorithm to achieve
the optimal transmission strategies with only private and incomplete
information. An intelligent SU performs Q-function updates based on the
conjecture over the other SUs' stochastic behaviors. This learning algorithm
provably converges given certain restrictions that arise during learning
procedure. Simulation experiments are used to verify the performance of our
algorithm and demonstrate its effectiveness of improving the energy efficiency
The Price of Governance: A Middle Ground Solution to Coordination in Organizational Control
Achieving coordination is crucial in organizational control. This paper
investigates a middle ground solution between decentralized interactions and
centralized administrations for coordinating agents beyond inefficient
behavior. We first propose the price of governance (PoG) to evaluate how such a
middle ground solution performs in terms of effectiveness and cost. We then
propose a hierarchical supervision framework to explicitly model the PoG, and
define step by step how to realize the core principle of the framework and
compute the optimal PoG for a control problem. Two illustrative case studies
are carried out to exemplify the applications of the proposed framework and its
methodology. Results show that by properly formulating and implementing each
step, the hierarchical supervision framework is capable of promoting
coordination among agents while bounding administrative cost to a minimum in
different kinds of organizational control problems
A Collaborative Multi-agent Reinforcement Learning Anti-jamming Algorithm in Wireless Networks
In this letter, we investigate the anti-jamming defense problem in multi-user
scenarios, where the coordination among users is taken into consideration. The
Markov game framework is employed to model and analyze the anti-jamming defense
problem, and a collaborative multi-agent anti-jamming algorithm (CMAA) is
proposed to obtain the optimal anti-jamming strategy. In sweep jamming
scenarios, on the one hand, the proposed CMAA can tackle the external malicious
jamming. On the other hand, it can effectively cope with the mutual
interference among users. Simulation results show that the proposed CMAA is
superior to both sensing based method and independent Q-learning method, and
has the highest normalized rate.Comment: 4 pages, 6 figures, Submitted to IEEE Wireless Communications Letter
Data Management in Industry 4.0: State of the Art and Open Challenges
Information and communication technologies are permeating all aspects of
industrial and manufacturing systems, expediting the generation of large
volumes of industrial data. This article surveys the recent literature on data
management as it applies to networked industrial environments and identifies
several open research challenges for the future. As a first step, we extract
important data properties (volume, variety, traffic, criticality) and identify
the corresponding data enabling technologies of diverse fundamental industrial
use cases, based on practical applications. Secondly, we provide a detailed
outline of recent industrial architectural designs with respect to their data
management philosophy (data presence, data coordination, data computation) and
the extent of their distributiveness. Then, we conduct a holistic survey of the
recent literature from which we derive a taxonomy of the latest advances on
industrial data enabling technologies and data centric services, spanning all
the way from the field level deep in the physical deployments, up to the cloud
and applications level. Finally, motivated by the rich conclusions of this
critical analysis, we identify interesting open challenges for future research.
The concepts presented in this article thematically cover the largest part of
the industrial automation pyramid layers. Our approach is multidisciplinary, as
the selected publications were drawn from two fields; the communications,
networking and computation field as well as the industrial, manufacturing and
automation field. The article can help the readers to deeply understand how
data management is currently applied in networked industrial environments, and
select interesting open research opportunities to pursue
Application of Machine Learning in Wireless Networks: Key Techniques and Open Issues
As a key technique for enabling artificial intelligence, machine learning
(ML) is capable of solving complex problems without explicit programming.
Motivated by its successful applications to many practical tasks like image
recognition, both industry and the research community have advocated the
applications of ML in wireless communication. This paper comprehensively
surveys the recent advances of the applications of ML in wireless
communication, which are classified as: resource management in the MAC layer,
networking and mobility management in the network layer, and localization in
the application layer. The applications in resource management further include
power control, spectrum management, backhaul management, cache management,
beamformer design and computation resource management, while ML based
networking focuses on the applications in clustering, base station switching
control, user association and routing. Moreover, literatures in each aspect is
organized according to the adopted ML techniques. In addition, several
conditions for applying ML to wireless communication are identified to help
readers decide whether to use ML and which kind of ML techniques to use, and
traditional approaches are also summarized together with their performance
comparison with ML based approaches, based on which the motivations of surveyed
literatures to adopt ML are clarified. Given the extensiveness of the research
area, challenges and unresolved issues are presented to facilitate future
studies, where ML based network slicing, infrastructure update to support ML
based paradigms, open data sets and platforms for researchers, theoretical
guidance for ML implementation and so on are discussed.Comment: 34 pages,8 figure
- …