218 research outputs found
Applications of Deep Reinforcement Learning in Communications and Networking: A Survey
This paper presents a comprehensive literature review on applications of deep
reinforcement learning in communications and networking. Modern networks, e.g.,
Internet of Things (IoT) and Unmanned Aerial Vehicle (UAV) networks, become
more decentralized and autonomous. In such networks, network entities need to
make decisions locally to maximize the network performance under uncertainty of
network environment. Reinforcement learning has been efficiently used to enable
the network entities to obtain the optimal policy including, e.g., decisions or
actions, given their states when the state and action spaces are small.
However, in complex and large-scale networks, the state and action spaces are
usually large, and the reinforcement learning may not be able to find the
optimal policy in reasonable time. Therefore, deep reinforcement learning, a
combination of reinforcement learning with deep learning, has been developed to
overcome the shortcomings. In this survey, we first give a tutorial of deep
reinforcement learning from fundamental concepts to advanced models. Then, we
review deep reinforcement learning approaches proposed to address emerging
issues in communications and networking. The issues include dynamic network
access, data rate control, wireless caching, data offloading, network security,
and connectivity preservation which are all important to next generation
networks such as 5G and beyond. Furthermore, we present applications of deep
reinforcement learning for traffic routing, resource sharing, and data
collection. Finally, we highlight important challenges, open issues, and future
research directions of applying deep reinforcement learning.Comment: 37 pages, 13 figures, 6 tables, 174 reference paper
Application of Machine Learning in Wireless Networks: Key Techniques and Open Issues
As a key technique for enabling artificial intelligence, machine learning
(ML) is capable of solving complex problems without explicit programming.
Motivated by its successful applications to many practical tasks like image
recognition, both industry and the research community have advocated the
applications of ML in wireless communication. This paper comprehensively
surveys the recent advances of the applications of ML in wireless
communication, which are classified as: resource management in the MAC layer,
networking and mobility management in the network layer, and localization in
the application layer. The applications in resource management further include
power control, spectrum management, backhaul management, cache management,
beamformer design and computation resource management, while ML based
networking focuses on the applications in clustering, base station switching
control, user association and routing. Moreover, literatures in each aspect is
organized according to the adopted ML techniques. In addition, several
conditions for applying ML to wireless communication are identified to help
readers decide whether to use ML and which kind of ML techniques to use, and
traditional approaches are also summarized together with their performance
comparison with ML based approaches, based on which the motivations of surveyed
literatures to adopt ML are clarified. Given the extensiveness of the research
area, challenges and unresolved issues are presented to facilitate future
studies, where ML based network slicing, infrastructure update to support ML
based paradigms, open data sets and platforms for researchers, theoretical
guidance for ML implementation and so on are discussed.Comment: 34 pages,8 figure
Decentralized Computation Offloading for Multi-User Mobile Edge Computing: A Deep Reinforcement Learning Approach
Mobile edge computing (MEC) emerges recently as a promising solution to
relieve resource-limited mobile devices from computation-intensive tasks, which
enables devices to offload workloads to nearby MEC servers and improve the
quality of computation experience. Nevertheless, by considering a MEC system
consisting of multiple mobile users with stochastic task arrivals and wireless
channels in this paper, the design of computation offloading policies is
challenging to minimize the long-term average computation cost in terms of
power consumption and buffering delay. A deep reinforcement learning (DRL)
based decentralized dynamic computation offloading strategy is investigated to
build a scalable MEC system with limited feedback. Specifically, a continuous
action space-based DRL approach named deep deterministic policy gradient (DDPG)
is adopted to learn efficient computation offloading policies independently at
each mobile user. Thus, powers of both local execution and task offloading can
be adaptively allocated by the learned policies from each user's local
observation of the MEC system. Numerical results are illustrated to demonstrate
that efficient policies can be learned at each user, and performance of the
proposed DDPG based decentralized strategy outperforms the conventional deep
Q-network (DQN) based discrete power control strategy and some other greedy
strategies with reduced computation cost. Besides, the power-delay tradeoff is
also analyzed for both the DDPG based and DQN based strategies
Deep Reinforcement Learning for Autonomous Internet of Things: Model, Applications and Challenges
The Internet of Things (IoT) extends the Internet connectivity into billions
of IoT devices around the world, where the IoT devices collect and share
information to reflect status of the physical world. The Autonomous Control
System (ACS), on the other hand, performs control functions on the physical
systems without external intervention over an extended period of time. The
integration of IoT and ACS results in a new concept - autonomous IoT (AIoT).
The sensors collect information on the system status, based on which the
intelligent agents in the IoT devices as well as the Edge/Fog/Cloud servers
make control decisions for the actuators to react. In order to achieve
autonomy, a promising method is for the intelligent agents to leverage the
techniques in the field of artificial intelligence, especially reinforcement
learning (RL) and deep reinforcement learning (DRL) for decision making. In
this paper, we first provide a tutorial of DRL, and then propose a general
model for the applications of RL/DRL in AIoT. Next, a comprehensive survey of
the state-of-art research on DRL for AIoT is presented, where the existing
works are classified and summarized under the umbrella of the proposed general
DRL model. Finally, the challenges and open issues for future research are
identified
Deep Reinforcement Learning for Backscatter-Aided Data Offloading in Mobile Edge Computing
Wireless network optimization has been becoming very challenging as the
problem size and complexity increase tremendously, due to close couplings among
network entities with heterogeneous service and resource requirements. By
continuously interacting with the environment, deep reinforcement learning
(DRL) provides a mechanism for different network entities to build knowledge
and make autonomous decisions to improve network performance. In this article,
we first review typical DRL approaches and recent enhancements. We then discuss
the applications of DRL for mobile edge computing (MEC), which can be used for
the low-power IoT devices, e.g., wireless sensors in healthcare monitoring, to
offload computation workload to nearby MEC servers. To balance power
consumption in offloading and computation, we propose a novel hybrid offloading
model that exploits the complement operations of RF communications and
low-power backscatter communications. The DRL framework is then customized to
optimize the transmission scheduling and workload allocation in two
communications technologies, which is shown to enhance the offloading
performance significantly compared with existing schemes.Comment: 15 pages, 6 figures, 15 reference
Vehicular Edge Computing via Deep Reinforcement Learning
The smart vehicles construct Vehicle of Internet which can execute various
intelligent services. Although the computation capability of the vehicle is
limited, multi-type of edge computing nodes provide heterogeneous resources for
vehicular services.When offloading the complicated service to the vehicular
edge computing node, the decision should consider numerous factors.The
offloading decision work mostly formulate the decision to a resource scheduling
problem with single or multiple objective function and some constraints, and
explore customized heuristics algorithms. However, offloading multiple data
dependency tasks in a service is a difficult decision, as an optimal solution
must understand the resource requirement, the access network, the user
mobility, and importantly the data dependency. Inspired by recent advances in
machine learning, we propose a knowledge driven (KD) service offloading
decision framework for Vehicle of Internet, which provides the optimal policy
directly from the environment. We formulate the offloading decision of
multi-task in a service as a long-term planning problem, and explores the
recent deep reinforcement learning to obtain the optimal solution. It considers
the future data dependency of the following tasks when making decision for a
current task from the learned offloading knowledge. Moreover, the framework
supports the pre-training at the powerful edge computing node and continually
online learning when the vehicular service is executed, so that it can adapt
the environment changes and learns policy that are sensible in hindsight. The
simulation results show that KD service offloading decision converges quickly,
adapts to different conditions, and outperforms the greedy offloading decision
algorithm.Comment: Preliminary report of ongoing wor
Edge Intelligence for Energy-efficient Computation Offloading and Resource Allocation in 5G Beyond
5G beyond is an end-edge-cloud orchestrated network that can exploit
heterogeneous capabilities of the end devices, edge servers, and the cloud and
thus has the potential to enable computation-intensive and delay-sensitive
applications via computation offloading. However, in multi user wireless
networks, diverse application requirements and the possibility of various radio
access modes for communication among devices make it challenging to design an
optimal computation offloading scheme. In addition, having access to complete
network information that includes variables such as wireless channel state, and
available bandwidth and computation resources, is a major issue. Deep
Reinforcement Learning (DRL) is an emerging technique to address such an issue
with limited and less accurate network information. In this paper, we utilize
DRL to design an optimal computation offloading and resource allocation
strategy for minimizing system energy consumption. We first present a
multi-user end-edge-cloud orchestrated network where all devices and base
stations have computation capabilities. Then, we formulate the joint
computation offloading and resource allocation problem as a Markov Decision
Process (MDP) and propose a new DRL algorithm to minimize system energy
consumption. Numerical results based on a real-world dataset demonstrate that
the proposed DRL-based algorithm significantly outperforms the benchmark
policies in terms of system energy consumption. Extensive simulations show that
learning rate, discount factor, and number of devices have considerable
influence on the performance of the proposed algorithm
Enhancing the performance of energy harvesting wireless communications using optimization and machine learning
The motivation behind this thesis is to provide efficient solutions for energy harvesting communications. Firstly, an energy harvesting underlay cognitive radio relaying network is investigated. In this context, the secondary network is an energy harvesting network. Closed-form expressions are derived for transmission power of secondary source and relay that maximizes the secondary network throughput. Secondly, a practical scenario in terms of information availability about the environment is investigated. We consider a communications system with a source capable of harvesting solar energy. Two cases are considered based on the knowledge availability about the underlying processes. When this knowledge is available, an algorithm using this knowledge is designed to maximize the expected throughput, while reducing the complexity of traditional methods. For the second case, when the knowledge about the underlying processes is unavailable, reinforcement learning is used. Thirdly, a number of learning architectures for reinforcement learning are introduced. They are called selector-actor-critic, tuner-actor-critic, and estimator-selector-actor-critic. The goal of the selector-actor-critic architecture is to increase the speed and the efficiency of learning an optimal policy by approximating the most promising action at the current state. The tuner-actor-critic aims at improving the learning process by providing the actor with a more accurate estimation about the value function. Estimator-selector-actor-critic is introduced to support intelligent agents. This architecture mimics rational humans in the way of analyzing available information, and making decisions. Then, a harvesting communications system working in an unknown environment is evaluated when it is supported by the proposed architectures. Fourthly, a realistic energy harvesting communications system is investigated. The state and action spaces of the underlying Markov decision process are continuous. Actor-critic is used to optimize the system performance. The critic uses a neural network to approximate the action-value function. The actor uses policy gradient to optimize the policy\u27s parameters to maximize the throughput
Deep Reinforcement Learning for Stochastic Computation Offloading in Digital Twin Networks
The rapid development of Industrial Internet of Things (IIoT) requires
industrial production towards digitalization to improve network efficiency.
Digital Twin is a promising technology to empower the digital transformation of
IIoT by creating virtual models of physical objects. However, the provision of
network efficiency in IIoT is very challenging due to resource-constrained
devices, stochastic tasks, and resources heterogeneity. Distributed resources
in IIoT networks can be efficiently exploited through computation offloading to
reduce energy consumption while enhancing data processing efficiency. In this
paper, we first propose a new paradigm Digital Twin Networks (DTN) to build
network topology and the stochastic task arrival model in IIoT systems. Then,
we formulate the stochastic computation offloading and resource allocation
problem to minimize the long-term energy efficiency. As the formulated problem
is a stochastic programming problem, we leverage Lyapunov optimization
technique to transform the original problem into a deterministic per-time slot
problem. Finally, we present Asynchronous Actor-Critic (AAC) algorithm to find
the optimal stochastic computation offloading policy. Illustrative results
demonstrate that our proposed scheme is able to significantly outperforms the
benchmarks.Comment: 10 page
Com-DDPG: A Multiagent Reinforcement Learning-based Offloading Strategy for Mobile Edge Computing
The development of mobile services has impacted a variety of
computation-intensive and time-sensitive applications, such as recommendation
systems and daily payment methods. However, computing task competition
involving limited resources increases the task processing latency and energy
consumption of mobile devices, as well as time constraints. Mobile edge
computing (MEC) has been widely used to address these problems. However, there
are limitations to existing methods used during computation offloading. On the
one hand, they focus on independent tasks rather than dependent tasks. The
challenges of task dependency in the real world, especially task segmentation
and integration, remain to be addressed. On the other hand, the multiuser
scenarios related to resource allocation and the mutex access problem must be
considered. In this paper, we propose a novel offloading approach, Com-DDPG,
for MEC using multiagent reinforcement learning to enhance the offloading
performance. First, we discuss the task dependency model, task priority model,
energy consumption model, and average latency from the perspective of server
clusters and multidependence on mobile tasks. Our method based on these models
is introduced to formalize communication behavior among multiple agents; then,
reinforcement learning is executed as an offloading strategy to obtain the
results. Because of the incomplete state information, long short-term memory
(LSTM) is employed as a decision-making tool to assess the internal state.
Moreover, to optimize and support effective action, we consider using a
bidirectional recurrent neural network (BRNN) to learn and enhance features
obtained from agents' communication. Finally, we simulate experiments on the
Alibaba cluster dataset. The results show that our method is better than other
baselines in terms of energy consumption, load status and latency
- …