A Reinforcement Learning-Based User-Assisted Caching Strategy for Dynamic Content Library in Small Cell Networks

This paper studies the problem of joint edge cache placement and content delivery in cache-enabled small cell networks in the presence of spatio-temporal content dynamics unknown a priori. The small base stations (SBSs) satisfy users’ content requests either directly from their local caches, or by retrieving from other SBSs’ caches or from the content server. In contrast to previous approaches that assume a static content library at the server, this paper considers a more realistic non-stationary content library, where new contents may emerge over time at different locations. To keep track of spatio-temporal content dynamics, we propose that the new contents cached at users can be exploited by the SBSs to timely update their flexible cache memories in addition to their routine off-peak main cache updates from the content server. To take into account the variations in traffic demands as well as the limited caching space at the SBSs, a user-assisted caching strategy is proposed based on reinforcement learning principles to progressively optimize the caching policy with the target of maximizing the weighted network utility in the long run. Simulation results verify the superior performance of the proposed caching strategy against various benchmark designs

Two Time-Scale Caching Placement and User Association in Dynamic Cellular Networks

With the rapid growth of data traffic in cellular networks, edge caching has become an emerging technology for traffic offloading. We investigate the caching placement and content delivery in cache-enabling cellular networks. To cope with the time-varying content popularity and user location in practical scenarios, we formulate a long-term joint dynamic optimization problem of caching placement and user association for minimizing the content delivery delay which considers both content transmission delay and content update delay. To solve this challenging problem, we decompose the optimization problem into two sub-problems, the user association sub-problem in a short time scale and the caching placement in a long time scale. Specifically, we propose a low complexity user association algorithm for a given caching placement in the short time scale. Then we develop a deep deterministic policy gradient based caching placement algorithm which involves the short time-scale user association decisions in the long time scale. Finally, we propose a joint user association and caching placement algorithm to obtain a sub-optimal solution for the proposed problem. We illustrate the convergence and performance of the proposed algorithm by simulation results. Simulation results show that compared with the benchmark algorithms, the proposed algorithm reduces the long-term content delivery delay in dynamic networks effectively

Reinforcement learning for proactive content caching in wireless networks

Proactive content caching (PC) at the edge of wireless networks, that is, at the base stations (BSs) and/or user equipments (UEs), is a promising strategy to successfully handle the ever-growing mobile data traffic and to improve the quality-of-service for content delivery over wireless networks. However, factors such as limitations in storage capacity, time-variations in wireless channel conditions as well as in content demand profile pose challenges that need to be addressed in order to realise the benefits of PC at the wireless edge. This thesis aims to develop PC solutions that address these challenges. We consider PC directly at UEs equipped with finite capacity cache memories. This consideration is done within the framework of a dynamic system, where mobile users randomly request contents from a non-stationary content library; new contents are added to the library over time and each content may remain in the library for a random lifetime within which it may be requested. Contents are delivered through wireless channels with time-varying quality, and any time contents are transmitted, a transmission cost associated with the number of bits downloaded and the channel quality of the receiving user(s) at that time is incurred by the system. We formulate each considered problem as a Markov decision process with the objective of minimising the long term expected average cost on the system. We then use reinforcement learning (RL) to solve this highly challenging problem with a prohibitively large state and action spaces. In particular, we employ policy approximation techniques for compact representation of complex policy structures, and policy gradient RL methods to train the system. In a single-user problem setting that we consider, we show the optimality of a threshold-based PC scheme that is adaptive to system dynamics. We use this result to characterise and design a multicast-aware PC scheme, based on deep RL framework, when we consider a multi-user problem setting. We perform extensive numerical simulations of the schemes we propose. Our results show not only significant improvements against the state-of-the-art reactive content delivery approaches, but also near-optimality of the proposed RL solutions based on comparisons with some lower bounds.Open Acces

Dynamic Resource Management in Integrated NOMA Terrestrial-Satellite Networks using Multi-Agent Reinforcement Learning

This study introduces a resource allocation framework for integrated satellite-terrestrial networks to address these challenges. The framework leverages local cache pool deployments and non-orthogonal multiple access (NOMA) to reduce time delays and improve energy efficiency. Our proposed approach utilizes a multi-agent enabled deep deterministic policy gradient algorithm (MADDPG) to optimize user association, cache design, and transmission power control, resulting in enhanced energy efficiency. The approach comprises two phases: User Association and Power Control, where users are treated as agents, and Cache Optimization, where the satellite (Bs) is considered the agent. Through extensive simulations, we demonstrate that our approach surpasses conventional single-agent deep reinforcement learning algorithms in addressing cache design and resource allocation challenges in integrated terrestrial-satellite networks. Specifically, our proposed approach achieves significantly higher energy efficiency and reduced time delays compared to existing methods.Comment: 16, 1

Deep learning-based edge caching for multi-cluster heterogeneous networks

© 2019, Springer-Verlag London Ltd., part of Springer Nature. In this work, we consider a time and space evolution cache refreshing in multi-cluster heterogeneous networks. We consider a two-step content placement probability optimization. At the initial complete cache refreshing optimization, the joint optimization of the activated base station density and the content placement probability is considered. And we transform this optimization problem into a GP problem. At the following partial cache refreshing optimization, we take the time–space evolution into consideration and derive a convex optimization problem subjected to the cache capacity constraint and the backhaul limit constraint. We exploit the redundant information in different content popularity using the deep neural network to avoid the repeated calculation because of the change in content popularity distribution at different time slots. Trained DNN can provide online response to content placement in a multi-cluster HetNet model instantaneously. Numerical results demonstrate the great approximation to the optimum and generalization ability