Learning Automata Based Q-Learning for Content Placement in Cooperative Caching

Author's accepted manuscript.© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.acceptedVersio

Achieving Fair Load Balancing by Invoking a Learning Automata-based Two Time Scale Separation Paradigm

Author's accepted manuscript.© 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.In this article, we consider the problem of load balancing (LB), but, unlike the approaches that have been proposed earlier, we attempt to resolve the problem in a fair manner (or rather, it would probably be more appropriate to describe it as an ε-fair manner because, although the LB can, probably, never be totally fair, we achieve this by being ``as close to fair as possible''). The solution that we propose invokes a novel stochastic learning automaton (LA) scheme, so as to attain a distribution of the load to a number of nodes, where the performance level at the different nodes is approximately equal and each user experiences approximately the same Quality of the Service (QoS) irrespective of which node that he/she is connected to. Since the load is dynamically varying, static resource allocation schemes are doomed to underperform. This is further relevant in cloud environments, where we need dynamic approaches because the available resources are unpredictable (or rather, uncertain) by virtue of the shared nature of the resource pool. Furthermore, we prove here that there is a coupling involving LA's probabilities and the dynamics of the rewards themselves, which renders the environments to be nonstationary. This leads to the emergence of the so-called property of ``stochastic diminishing rewards.'' Our newly proposed novel LA algorithm ε-optimally solves the problem, and this is done by resorting to a two-time-scale-based stochastic learning paradigm. As far as we know, the results presented here are of a pioneering sort, and we are unaware of any comparable results.acceptedVersio

Privacy-preserving federated deep learning for cooperative hierarchical caching in fog computing

Over the past few years, Fog Radio Access Networks (F-RANs) have become a promising paradigm to support the tremendously increasing demands of multimedia services, by pushing computation and storage functionalities towards the edge of networks, closer to users. In F-RANs, distributed edge caching among Fog Access Points (F-APs) can effectively reduce network traffic and service latency as it places popular contents at local caches of F-APs rather than the remote cloud. Due to the limited caching resources of F-APs and spatio-temporally fluctuant content demands from users, many cooperative caching schemes were designed to decide which contents are popular and how to cache them. However, these approaches often collect and analyse the data from Internet-of-Things (IoT) devices at a central server to predict the content popularity for caching, which raises serious privacy issues. To tackle this challenge, we propose a Federated Learning based Cooperative Hierarchical Caching scheme (FLCH), which keeps data locally and employs IoT devices to train a shared learning model for content popularity prediction. FLCH exploits horizontal cooperation between neighbour F-APs and vertical cooperation between the BaseBand Unit (BBU) pool and F-APs to cache contents with different degrees of popularity. Moreover, FLCH integrates a differential privacy mechanism to achieve a strict privacy guarantee. Experimental results demonstrate that FLCH outperforms five important baseline schemes in terms of the cache hit ratio, while preserving data privacy. Moreover, the results show the effectiveness of the proposed cooperative hierarchical caching mechanism for FLCH

Agile Cache Replacement in Edge Computing via Offline-Online Deep Reinforcement Learning

This is the author accepted manuscript. The final version is available from IEEE via the DOI in this recordOne fundamental problem of content caching in edge computing is how to replace contents in edge servers with limited capacities to meet the dynamic requirements of users without knowing their preferences in advance. Recently, online deep reinforcement learning (DRL)-based caching methods have been developed to address this problem by learning an edge cache replacement policy using samples collected from continuous interactions (trial and error) with the environment. However, in practice, the online data collection phase is often expensive and time-consuming, thus hindering the practical deployment of online DRL-based methods. To bridge this gap, we propose a novel Agile edge Cache replacement method based on Offline-online deep Reinforcement learNing (ACORN), which can efficiently learn an edge cache replacement policy offline from a training dataset collected by a behavior policy (e.g., Least Recently Used) and then improve it with fast online fine-tuning. We also design a specific convolutional neural network structure with multiple branches to effectively extract content popularity knowledge from the dataset. Experimental results show that the offline policy generated by ACORN outperforms the behavior policy by up to 38%. Through online fine-tuning, ACORN also achieves the number of cache hits as good as that of several advanced DRL-based methods while significantly reducing the number of training epochs by up to 40%.UKRIHorizon Europ

User Preferences-Based Proactive Content Caching with Characteristics Differentiation in HetNets

With the proliferation of mobile applications, the explosion of mobile data traffic imposes a significant burden on backhaul links with limited capacity in heterogeneous cellular networks (HetNets). To alleviate this challenge, content caching based on popularity at Small Base Stations (SBSs) has emerged as a promising solution. However, accurately predicting the file popularity profile for SBSs remains a key challenge due to variations in content characteristics and user preferences. Moreover, factors such as content size and the length of time slots (that is, the time duration of the update cycle for SBSs) critically impact the performance of caching schemes with limited storage capacity. In this paper, a realism-oriented intelligent caching (RETINA) is proposed to address the problem of content caching with unknown file popularity profiles, considering varying content sizes and time slots lengths. Our simulation results demonstrate that RETINA can significantly enhance the cache hit rate by 4%–12% compared to existing content caching schemes

On the Theory and Applications of Hierarchical Learning Automata and Object Migration Automata

Paper III, IV and VIII are excluded due to copyright.The paradigm of Artificial Intelligence (AI) and the sub-group of Machine Learning (ML) have attracted exponential interest in our society in recent years. The domain of ML contains numerous methods, and it is desirable (and in one sense, mandatory) that these methods are applicable and valuable to real-life challenges. Learning Automata (LA) is an intriguing and classical direction within ML. In LA, non-human agents can find optimal solutions to various problems through the concept of learning. The LA instances learn through Agent-Environment interactions, where advantageous behavior is rewarded, and disadvantageous behavior is penalized. Consequently, the agent eventually, and hopefully, learns the optimal action from a set of actions. LA has served as a foundation for Reinforcement Learning (RL), and the field of LA has been studied for decades. However, many improvements can still be made to render these algorithms to be even more convenient and effective. In this dissertation, we record our research contributions to the design and applications within the field of LA. Our research includes novel improvements to the domain of hierarchical LA, major advancements to the family of Object Migration Automata (OMA) algorithms, and a novel application of LA, where it was utilized to solve challenges in a mobile radio communication system. More specifically, we introduced the novel Hierarchical Discrete Pursuit Automaton (HDPA), which significantly improved the state of the art in terms of effectiveness for problems with high accuracy requirements, and we mathematically proved its ϵ-optimality. In addition, we proposed the Action Distribution Enhanced (ADE) approach to hierarchical LA schemes which significantly reduced the number of iterations required before the machines reached convergence. The existing schemes in the OMA paradigm, which are able to solve partitioning problems, could only solve problems with equally-sized partitions. Therefore, we proposed two novel methods that could handle unequally-sized partitions. In addition, we rigorously summarized the OMA domain, outlined its potential benefits to society, and listed further development cases for future researchers in the field. With regard to applications, we proposed an OMA-based approach to the grouping and power allocation in Non-orthogonal Multiple Access (NOMA) systems, demonstrating the applicability of the OMA and its advantage in solving fairly complicated stochastic problems. The details of these contributions and their published scientific impacts will be summarized in this dissertation, before we present some of the research contributions in their entirety.publishedVersio