51 research outputs found
Decentralized Microgrid Energy Management: A Multi-agent Correlated Q-learning Approach
Microgrids (MG) are anticipated to be important players in the future smart
grid. For proper operation of MGs an Energy Management System (EMS) is
essential. The EMS of an MG could be rather complicated when renewable energy
resources (RER), energy storage system (ESS) and demand side management (DSM)
need to be orchestrated. Furthermore, these systems may belong to different
entities and competition may exist between them. Nash equilibrium is most
commonly used for coordination of such entities however the convergence and
existence of Nash equilibrium can not always be guaranteed. To this end, we use
the correlated equilibrium to coordinate agents, whose convergence can be
guaranteed. In this paper, we build an energy trading model based on mid-market
rate, and propose a correlated Q-learning (CEQ) algorithm to maximize the
revenue of each agent. Our results show that CEQ is able to balance the revenue
of agents without harming total benefit. In addition, compared with Q-learning
without correlation, CEQ could save 19.3% cost for the DSM agent and 44.2% more
benefits for the ESS agent.Comment: Accepted by 2020 IEEE International Conference on SmartGridComm,
978-1-7281-6127-3/20/$31.00 copyright 2020 IEE
Correlated Deep Q-learning based Microgrid Energy Management
Microgrid (MG) energy management is an important part of MG operation.
Various entities are generally involved in the energy management of an MG,
e.g., energy storage system (ESS), renewable energy resources (RER) and the
load of users, and it is crucial to coordinate these entities. Considering the
significant potential of machine learning techniques, this paper proposes a
correlated deep Q-learning (CDQN) based technique for the MG energy management.
Each electrical entity is modeled as an agent which has a neural network to
predict its own Q-values, after which the correlated Q-equilibrium is used to
coordinate the operation among agents. In this paper, the Long Short Term
Memory networks (LSTM) based deep Q-learning algorithm is introduced and the
correlated equilibrium is proposed to coordinate agents. The simulation result
shows 40.9% and 9.62% higher profit for ESS agent and photovoltaic (PV) agent,
respectively.Comment: Accepted by 2020 IEEE 25th International Workshop on CAMAD,
978-1-7281-6339-0/20/$31.00 \copyright 2020 IEE
Joint Sensing and Communications for Deep Reinforcement Learning-based Beam Management in 6G
User location is a piece of critical information for network management and
control. However, location uncertainty is unavoidable in certain settings
leading to localization errors. In this paper, we consider the user location
uncertainty in the mmWave networks, and investigate joint vision-aided sensing
and communications using deep reinforcement learning-based beam management for
future 6G networks. In particular, we first extract pixel characteristic-based
features from satellite images to improve localization accuracy. Then we
propose a UK-medoids based method for user clustering with location
uncertainty, and the clustering results are consequently used for the beam
management. Finally, we apply the DRL algorithm for intra-beam radio resource
allocation. The simulations first show that our proposed vision-aided method
can substantially reduce the localization error. The proposed UK-medoids and
DRL based scheme (UKM-DRL) is compared with two other schemes: K-means based
clustering and DRL based resource allocation (K-DRL) and UK-means based
clustering and DRL based resource allocation (UK-DRL). The proposed method has
17.2% higher throughput and 7.7% lower delay than UK-DRL, and more than doubled
throughput and 55.8% lower delay than K-DRL
Learning from Peers: Deep Transfer Reinforcement Learning for Joint Radio and Cache Resource Allocation in 5G RAN Slicing
Radio access network (RAN) slicing is an important pillar in cross-domain
network slicing which covers RAN, edge, transport and core slicing. The
evolving network architecture requires the orchestration of multiple network
resources such as radio and cache resources. In recent years, machine learning
(ML) techniques have been widely applied for network management. However, most
existing works do not take advantage of the knowledge transfer capability in
ML. In this paper, we propose a deep transfer reinforcement learning (DTRL)
scheme for joint radio and cache resource allocation to serve 5G RAN slicing.
We first define a hierarchical architecture for the joint resource allocation.
Then we propose two DTRL algorithms: Q-value-based deep transfer reinforcement
learning (QDTRL) and action selection-based deep transfer reinforcement
learning (ADTRL). In the proposed schemes, learner agents utilize expert
agents' knowledge to improve their performance on target tasks. The proposed
algorithms are compared with both the model-free exploration bonus deep
Q-learning (EB-DQN) and the model-based priority proportional fairness and
time-to-live (PPF-TTL) algorithms. Compared with EB-DQN, our proposed DTRL
based method presents 21.4% lower delay for Ultra Reliable Low Latency
Communications (URLLC) slice and 22.4% higher throughput for enhanced Mobile
Broad Band (eMBB) slice, while achieving significantly faster convergence than
EB-DQN. Moreover, 40.8% lower URLLC delay and 59.8% higher eMBB throughput are
observed with respect to PPF-TTL.Comment: Under review of IEEE Transactions on Cognitive Communications and
Networkin
Federated Deep Reinforcement Learning for Resource Allocation in O-RAN Slicing
Recently, open radio access network (O-RAN) has become a promising technology
to provide an open environment for network vendors and operators. Coordinating
the x-applications (xAPPs) is critical to increase flexibility and guarantee
high overall network performance in O-RAN. Meanwhile, federated reinforcement
learning has been proposed as a promising technique to enhance the
collaboration among distributed reinforcement learning agents and improve
learning efficiency. In this paper, we propose a federated deep reinforcement
learning algorithm to coordinate multiple independent xAPPs in O-RAN for
network slicing. We design two xAPPs, namely a power control xAPP and a
slice-based resource allocation xAPP, and we use a federated learning model to
coordinate two xAPP agents to enhance learning efficiency and improve network
performance. Compared with conventional deep reinforcement learning, our
proposed algorithm can achieve 11% higher throughput for enhanced mobile
broadband (eMBB) slices and 33% lower delay for ultra-reliable low-latency
communication (URLLC) slices
- …