Reinforcement learning for proactive content caching in wireless networks

Proactive content caching (PC) at the edge of wireless networks, that is, at the base stations (BSs) and/or user equipments (UEs), is a promising strategy to successfully handle the ever-growing mobile data traffic and to improve the quality-of-service for content delivery over wireless networks. However, factors such as limitations in storage capacity, time-variations in wireless channel conditions as well as in content demand profile pose challenges that need to be addressed in order to realise the benefits of PC at the wireless edge. This thesis aims to develop PC solutions that address these challenges. We consider PC directly at UEs equipped with finite capacity cache memories. This consideration is done within the framework of a dynamic system, where mobile users randomly request contents from a non-stationary content library; new contents are added to the library over time and each content may remain in the library for a random lifetime within which it may be requested. Contents are delivered through wireless channels with time-varying quality, and any time contents are transmitted, a transmission cost associated with the number of bits downloaded and the channel quality of the receiving user(s) at that time is incurred by the system. We formulate each considered problem as a Markov decision process with the objective of minimising the long term expected average cost on the system. We then use reinforcement learning (RL) to solve this highly challenging problem with a prohibitively large state and action spaces. In particular, we employ policy approximation techniques for compact representation of complex policy structures, and policy gradient RL methods to train the system. In a single-user problem setting that we consider, we show the optimality of a threshold-based PC scheme that is adaptive to system dynamics. We use this result to characterise and design a multicast-aware PC scheme, based on deep RL framework, when we consider a multi-user problem setting. We perform extensive numerical simulations of the schemes we propose. Our results show not only significant improvements against the state-of-the-art reactive content delivery approaches, but also near-optimality of the proposed RL solutions based on comparisons with some lower bounds.Open Acces

Towards Massive Machine Type Communications in Ultra-Dense Cellular IoT Networks: Current Issues and Machine Learning-Assisted Solutions

The ever-increasing number of resource-constrained Machine-Type Communication (MTC) devices is leading to the critical challenge of fulfilling diverse communication requirements in dynamic and ultra-dense wireless environments. Among different application scenarios that the upcoming 5G and beyond cellular networks are expected to support, such as eMBB, mMTC and URLLC, mMTC brings the unique technical challenge of supporting a huge number of MTC devices, which is the main focus of this paper. The related challenges include QoS provisioning, handling highly dynamic and sporadic MTC traffic, huge signalling overhead and Radio Access Network (RAN) congestion. In this regard, this paper aims to identify and analyze the involved technical issues, to review recent advances, to highlight potential solutions and to propose new research directions. First, starting with an overview of mMTC features and QoS provisioning issues, we present the key enablers for mMTC in cellular networks. Along with the highlights on the inefficiency of the legacy Random Access (RA) procedure in the mMTC scenario, we then present the key features and channel access mechanisms in the emerging cellular IoT standards, namely, LTE-M and NB-IoT. Subsequently, we present a framework for the performance analysis of transmission scheduling with the QoS support along with the issues involved in short data packet transmission. Next, we provide a detailed overview of the existing and emerging solutions towards addressing RAN congestion problem, and then identify potential advantages, challenges and use cases for the applications of emerging Machine Learning (ML) techniques in ultra-dense cellular networks. Out of several ML techniques, we focus on the application of low-complexity Q-learning approach in the mMTC scenarios. Finally, we discuss some open research challenges and promising future research directions.Comment: 37 pages, 8 figures, 7 tables, submitted for a possible future publication in IEEE Communications Surveys and Tutorial

Multi-Dimensional Resource Orchestration in Vehicular Edge Networks

In the era of autonomous vehicles, the advanced technologies of connected vehicle lead to the development of driving-related applications to meet the stringent safety requirements and the infotainment applications to improve passenger experience. Newly developed vehicular applications require high-volume data transmission, accurate sensing data collection, and reliable interaction, imposing substantial constrains on vehicular networks that solely rely on cellular networks to fetch data from the Internet and on-board processors to make driving decisions. To enhance multifarious vehicular applications, Heterogeneous Vehicular Networks (HVNets) have been proposed, in which edge nodes, including base stations and roadside units, can provide network connections, resulting in significantly reduced vehicular communication cost. In addition, caching servers are equipped at the edge nodes, to further alleviate the communication load for backhaul links and reduce data downloading delay. Hence, we aim to orchestrate the multi-dimensional resources, including communication, caching, and sensing resources, in the complex and dynamic vehicular environment to enhance vehicular edge network performance. The main technical issues are: 1) to accommodate the delivery services for both location-based and popular contents, the scheme of caching contents at edge servers should be devised, considering the cooperation of caching servers at different edge nodes, the mobility of vehicles, and the differential requirements of content downloading services; 2) to support the safety message exchange and collective perception services for vehicles, communication and sensing resources are jointly allocated, the decisions of which are coupled due to the resource sharing among different services and neighboring vehicles; and 3) for interaction-intensive service provisioning, e.g., trajectory design, the forwarding resources in core networks are allocated to achieve delay-sensitive packet transmissions between vehicles and management controllers, ensuring the high-quality interactivity. In this thesis, we design the multi-dimensional resource orchestration schemes in the edge assisted HVNets to address the three technical issues. Firstly, we design a cooperative edge caching scheme to support various vehicular content downloading services, which allows vehicles to fetch one content from multiple caching servers cooperatively. In particular, we consider two types of vehicular content requests, i.e., location-based and popular contents, with different delay requirements. Both types of contents are encoded according to fountain code and cooperatively cached at multiple servers. The proposed scheme can be optimized by finding an optimal cooperative content placement that determines the placing locations and proportions for all contents. To this end, we analyze the upper bound proportion of content caching at a single server and provide the respective theoretical analysis of transmission delay and service cost (including content caching and transmission cost) for both types of contents. We then formulate an optimization problem of cooperative content placement to minimize the overall transmission delay and service cost. As the problem is a multi-objective multi-dimensional multi-choice knapsack one, which is proved to be NP-hard, we devise an ant colony optimization-based scheme to solve the problem and achieve a near-optimal solution. Simulation results are provided to validate the performance of the proposed scheme, including its convergence and optimality of caching, while guaranteeing low transmission delay and service cost. Secondly, to support the vehicular safety message transmissions, we propose a two-level adaptive resource allocation (TARA) framework. In particular, three types of safety message are considered in urban vehicular networks, i.e., the event-triggered message for urgent condition warning, the periodic message for vehicular status notification, and the message for environmental perception. Roadside units are deployed for network management, and thus messages can be transmitted through either vehicle-to-infrastructure or vehicle-to-vehicle connections. To satisfy the requirements of different message transmissions, the proposed TARA framework consists of a group-level resource reservation module and a vehicle-level resource allocation module. Particularly, the resource reservation module is designed to allocate resources to support different types of message transmission for each vehicle group at the first level, and the group is formed by a set of neighboring vehicles. To learn the implicit relation between the resource demand and message transmission requests, a supervised learning model is devised in the resource reservation module, where to obtain the training data we further propose a sequential resource allocation (SRA) scheme. Based on historical network information, the SRA scheme offline optimizes the allocation of sensing resources (i.e., choosing vehicles to provide perception data) and communication resources. With the resource reservation result for each group, the vehicle-level resource allocation module is then devised to distribute specific resources for each vehicle to satisfy the differential requirements in real time. Extensive simulation results are provided to demonstrate the effectiveness of the proposed TARA framework in terms of the high successful reception ratio and low latency for message transmissions, and the high quality of collective environmental perception. Thirdly, we investigate forwarding resource sharing scheme to support interaction intensive services in HVNets, especially for the delay-sensitive packet transmission between vehicles and management controllers. A learning-based proactive resource sharing scheme is proposed for core communication networks, where the available forwarding resources at a switch are proactively allocated to the traffic flows in order to maximize the efficiency of resource utilization with delay satisfaction. The resource sharing scheme consists of two joint modules: estimation of resource demands and allocation of available resources. For service provisioning, resource demand of each traffic flow is estimated based on the predicted packet arrival rate. Considering the distinct features of each traffic flow, a linear regression scheme is developed for resource demand estimation, utilizing the mapping relation between traffic flow status and required resources, upon which a network switch makes decision on allocating available resources for delay satisfaction and efficient resource utilization. To learn the implicit relation between the allocated resources and delay, a multi-armed bandit learning-based resource sharing scheme is proposed, which enables fast resource sharing adjustment to traffic arrival dynamics. The proposed scheme is proved to be asymptotically approaching the optimal strategy, with polynomial time complexity. Extensive simulation results are presented to demonstrate the effectiveness of the proposed resource sharing scheme in terms of delay satisfaction, traffic adaptiveness, and resource sharing gain. In summary, we have investigated the cooperative caching placement for content downloading services, joint communication and sensing resource allocation for safety message transmissions, and forwarding resource sharing scheme in core networks for interaction intensive services. The schemes developed in the thesis should provide practical and efficient solutions to manage the multi-dimensional resources in vehicular networks

High-Performance Modelling and Simulation for Big Data Applications

This open access book was prepared as a Final Publication of the COST Action IC1406 “High-Performance Modelling and Simulation for Big Data Applications (cHiPSet)“ project. Long considered important pillars of the scientific method, Modelling and Simulation have evolved from traditional discrete numerical methods to complex data-intensive continuous analytical optimisations. Resolution, scale, and accuracy have become essential to predict and analyse natural and complex systems in science and engineering. When their level of abstraction raises to have a better discernment of the domain at hand, their representation gets increasingly demanding for computational and data resources. On the other hand, High Performance Computing typically entails the effective use of parallel and distributed processing units coupled with efficient storage, communication and visualisation systems to underpin complex data-intensive applications in distinct scientific and technical domains. It is then arguably required to have a seamless interaction of High Performance Computing with Modelling and Simulation in order to store, compute, analyse, and visualise large data sets in science and engineering. Funded by the European Commission, cHiPSet has provided a dynamic trans-European forum for their members and distinguished guests to openly discuss novel perspectives and topics of interests for these two communities. This cHiPSet compendium presents a set of selected case studies related to healthcare, biological data, computational advertising, multimedia, finance, bioinformatics, and telecommunications

A Gentle Introduction to Reinforcement Learning and its Application in Different Fields

Due to the recent progress in Deep Neural Networks, Reinforcement Learning (RL) has become one of the most important and useful technology. It is a learning method where a software agent interacts with an unknown environment, selects actions, and progressively discovers the environment dynamics. RL has been effectively applied in many important areas of real life. This article intends to provide an in-depth introduction of the Markov Decision Process, RL and its algorithms. Moreover, we present a literature review of the application of RL to a variety of fields, including robotics and autonomous control, communication and networking, natural language processing, games and self-organized system, scheduling management and configuration of resources, and computer vision

High-Performance Modelling and Simulation for Big Data Applications

This open access book was prepared as a Final Publication of the COST Action IC1406 “High-Performance Modelling and Simulation for Big Data Applications (cHiPSet)“ project. Long considered important pillars of the scientific method, Modelling and Simulation have evolved from traditional discrete numerical methods to complex data-intensive continuous analytical optimisations. Resolution, scale, and accuracy have become essential to predict and analyse natural and complex systems in science and engineering. When their level of abstraction raises to have a better discernment of the domain at hand, their representation gets increasingly demanding for computational and data resources. On the other hand, High Performance Computing typically entails the effective use of parallel and distributed processing units coupled with efficient storage, communication and visualisation systems to underpin complex data-intensive applications in distinct scientific and technical domains. It is then arguably required to have a seamless interaction of High Performance Computing with Modelling and Simulation in order to store, compute, analyse, and visualise large data sets in science and engineering. Funded by the European Commission, cHiPSet has provided a dynamic trans-European forum for their members and distinguished guests to openly discuss novel perspectives and topics of interests for these two communities. This cHiPSet compendium presents a set of selected case studies related to healthcare, biological data, computational advertising, multimedia, finance, bioinformatics, and telecommunications

Mobility-aware Software-Defined Service-Centric Networking for Service Provisioning in Urban Environments

Disruptive applications for mobile devices, such as the Internet of Things, Connected and Autonomous Vehicles, Immersive Media, and others, have requirements that the current Cloud Computing paradigm cannot meet. These unmet requirements bring the necessity to deploy geographically distributed computing architectures, such as Fog and Mobile Edge Computing. However, bringing computing close to users has its costs. One example of cost is the complexity introduced by the management of the mobility of the devices at the edge. This mobility may lead to issues, such as interruption of the communication with service instances hosted at the edge or an increase in communication latency during mobility events, e.g., handover. These issues, caused by the lack of mobility-aware service management solutions, result in degradation in service provisioning. The present thesis proposes a series of protocols and algorithms to handle user and service mobility at the edge of the network. User mobility is characterized when user change access points of wireless networks, while service mobility happens when services have to be provisioned from different hosts. It assembles them in a solution for mobility-aware service orchestration based on Information-Centric Networking (ICN) and runs on top of Software-Defined Networking (SDN). This solution addresses three issues related to handling user mobility at the edge: (i) proactive support for user mobility events, (ii) service instance addressing management, and (iii) distributed application state data management. For (i), we propose a proactive SDN-based handover scheme. For (ii), we propose an ICN addressing strategy to remove the necessity of updating addresses after service mobility events. For (iii), we propose a graph-based framework for state data placement in the network nodes that accounts for user mobility and latency requirements. The protocols and algorithms proposed in this thesis were compared with different approaches from the literature through simulation. Our results show that the proposed solution can reduce service interruption and latency in the presence of user and service mobility events while maintaining reasonable overhead costs regarding control messages sent in the network by the SDN controller