A comprehensive survey on reinforcement-learning-based computation offloading techniques in Edge Computing Systems

Producción CientíficaIn recent years, the number of embedded computing devices connected to the Internet has exponentially increased. At the same time, new applications are becoming more complex and computationally demanding, which can be a problem for devices, especially when they are battery powered. In this context, the concepts of computation offloading and edge computing, which allow applications to be fully or partially offloaded and executed on servers close to the devices in the network, have arisen and received increasing attention. Then, the design of algorithms to make the decision of which applications or tasks should be offloaded, and where to execute them, is crucial. One of the options that has been gaining momentum lately is the use of Reinforcement Learning (RL) and, in particular, Deep Reinforcement Learning (DRL), which enables learning optimal or near-optimal offloading policies adapted to each particular scenario. Although the use of RL techniques to solve the computation offloading problem in edge systems has been covered by some surveys, it has been done in a limited way. For example, some surveys have analysed the use of RL to solve various networking problems, with computation offloading being one of them, but not the primary focus. Other surveys, on the other hand, have reviewed techniques to solve the computation offloading problem, being RL just one of the approaches considered. To the best of our knowledge, this is the first survey that specifically focuses on the use of RL and DRL techniques for computation offloading in edge computing system. We present a comprehensive and detailed survey, where we analyse and classify the research papers in terms of use cases, network and edge computing architectures, objectives, RL algorithms, decision-making approaches, and time-varying characteristics considered in the analysed scenarios. In particular, we include a series of tables to help researchers identify relevant papers based on specific features, and analyse which scenarios and techniques are most frequently considered in the literature. Finally, this survey identifies a number of research challenges, future directions and areas for further study.Consejería de Educación de la Junta de Castilla y León y FEDER (VA231P20)Ministerio de Ciencia e Innovación y Agencia Estatal de Investigación (Proyecto PID2020-112675RB-C42, PID2021-124463OBI00 y RED2018-102585-T, financiados por MCIN/AEI/10.13039/501100011033

Deep reinforcement learning algorithms in multi agent changing environments using potential fields

Se propone el desarrollo de sistemas y algoritmos de aprendizaje reforzado profundo para entornos de vehículos autónomos. Para ello se propone inicialmente realizar una búsqueda bibliográfica sobre el uso de esta técnica de aprendizaje reforzado profundo para aplicaciones futuras de vehículos autónomos. Otro elemento básico de este proyecto será el desarrollo de herramientas de aprendizaje reforzado profundo, para mejorar en lo posible, la capacidad de aprendizaje del vehículo, la capacidad de adaptación a un entorno cambiante, y su capacidad final de decidir y realizar uThe project explores the possibilities offered by reinforcement learning in the field of robotics with the vision of guiding robots in changing environments with collision avoidance through potential fields. For this, the DDPG, TD3, SAC and PPO reinforcement learning algorithms are implemented through the Matlab Toolbox "Reinforcement Learning" with the aim of carrying out a comparative study on which of them is the most optimal for different configurations of environments and parameters, with the help of training graphs and statistical tables. Also, potential fields have been developed in this project, demonstrating to be a suitable tool for guiding robots in changing environments, and even to implement multi agent scenarios, avoiding collisions among them and enhancing collaboration.El projecte explora les possibilitats que ofereix l'aprenentatge per reforç en l'àmbit de la robòtica amb la visió de guiar robots a través d'entorns canviants amb evitació de col·lisions mitjançant camps de potencials. Per això s'implementen els algorismes d'aprenentatge per reforç DDPG, TD3, SAC i PPO per mitjà de la Toolbox de Matlab Reinforcement Learning amb l'objectiu de fer un estudi comparatiu sobre quin d'ells és el més òptim per a diferents configuracions d'entorns i paràmetres; tot això amb l'ajuda de gràfiques d'entrenament i taules estadístiques. Així mateix, s'han desenvolupat camps potencials en aquest projecte, demostrant ser una eina adequada per a guiar robots en entorns canviants, i fins i tot per implementar escenaris multiagent, evitant col·lisions entre ells i potenciant la col·laboració.El proyecto explora las posibilidades que ofrece el aprendizaje por refuerzo en el ámbito de la robótica con la visión de guiar a robots a través de entornos cambiantes con evitación de colisiones mediante campos potenciales. Para ello se implementan los algoritmos de aprendizaje por refuerzo DDPG, TD3, SAC y PPO por intermedio de la Toolbox de Matlab "Reinforcement Learning" con el objetivo de realizar un estudio comparativo sobre cuál de ellos es el más óptimo para diferentes configuraciones de entornos y parámetros; todo ello con la ayuda de gráficas de entrenamiento y tablas estadísticas. Además, en este proyecto se han desarrollado campos potenciales, demostrando ser una herramienta adecuada para guiar robots en entornos cambiantes, e incluso implementar escenarios multiagente, evitando colisiones entre ellos y potenciando la colaboración

Engineering Resilient Collective Adaptive Systems by Self-Stabilisation

Collective adaptive systems are an emerging class of networked computational systems, particularly suited in application domains such as smart cities, complex sensor networks, and the Internet of Things. These systems tend to feature large scale, heterogeneity of communication model (including opportunistic peer-to-peer wireless interaction), and require inherent self-adaptiveness properties to address unforeseen changes in operating conditions. In this context, it is extremely difficult (if not seemingly intractable) to engineer reusable pieces of distributed behaviour so as to make them provably correct and smoothly composable. Building on the field calculus, a computational model (and associated toolchain) capturing the notion of aggregate network-level computation, we address this problem with an engineering methodology coupling formal theory and computer simulation. On the one hand, functional properties are addressed by identifying the largest-to-date field calculus fragment generating self-stabilising behaviour, guaranteed to eventually attain a correct and stable final state despite any transient perturbation in state or topology, and including highly reusable building blocks for information spreading, aggregation, and time evolution. On the other hand, dynamical properties are addressed by simulation, empirically evaluating the different performances that can be obtained by switching between implementations of building blocks with provably equivalent functional properties. Overall, our methodology sheds light on how to identify core building blocks of collective behaviour, and how to select implementations that improve system performance while leaving overall system function and resiliency properties unchanged.Comment: To appear on ACM Transactions on Modeling and Computer Simulatio

How to Place Your Apps in the Fog -- State of the Art and Open Challenges

Fog computing aims at extending the Cloud towards the IoT so to achieve improved QoS and to empower latency-sensitive and bandwidth-hungry applications. The Fog calls for novel models and algorithms to distribute multi-service applications in such a way that data processing occurs wherever it is best-placed, based on both functional and non-functional requirements. This survey reviews the existing methodologies to solve the application placement problem in the Fog, while pursuing three main objectives. First, it offers a comprehensive overview on the currently employed algorithms, on the availability of open-source prototypes, and on the size of test use cases. Second, it classifies the literature based on the application and Fog infrastructure characteristics that are captured by available models, with a focus on the considered constraints and the optimised metrics. Finally, it identifies some open challenges in application placement in the Fog

Partitioning workflow applications over federated clouds to meet non-functional requirements

PhD ThesisWith cloud computing, users can acquire computer resources when they need them on a pay-as-you-go business model. Because of this, many applications are now being deployed in the cloud, and there are many di erent cloud providers worldwide. Importantly, all these various infrastructure providers o er services with di erent levels of quality. For example, cloud data centres are governed by the privacy and security policies of the country where the centre is located, while many organisations have created their own internal \private cloud" to meet security needs. With all this varieties and uncertainties, application developers who decide to host their system in the cloud face the issue of which cloud to choose to get the best operational conditions in terms of price, reliability and security. And the decision becomes even more complicated if their application consists of a number of distributed components, each with slightly di erent requirements. Rather than trying to identify the single best cloud for an application, this thesis considers an alternative approach, that is, combining di erent clouds to meet users' non-functional requirements. Cloud federation o ers the ability to distribute a single application across two or more clouds, so that the application can bene t from the advantages of each one of them. The key challenge for this approach is how to nd the distribution (or deployment) of application components, which can yield the greatest bene ts. In this thesis, we tackle this problem and propose a set of algorithms, and a framework, to partition a work ow-based application over federated clouds in order to exploit the strengths of each cloud. The speci c goal is to split a distributed application structured as a work ow such that the security and reliability requirements of each component are met, whilst the overall cost of execution is minimised. To achieve this, we propose and evaluate a cloud broker for partitioning a work ow application over federated clouds. The broker integrates with the e-Science Central cloud platform to automatically deploy a work ow over public and private clouds. We developed a deployment planning algorithm to partition a large work ow appli- - i - cation across federated clouds so as to meet security requirements and minimise the monetary cost. A more generic framework is then proposed to model, quantify and guide the partitioning and deployment of work ows over federated clouds. This framework considers the situation where changes in cloud availability (including cloud failure) arise during work ow execution

Data centre optimisation enhanced by software defined networking

Contemporary Cloud Computing infrastructures are being challenged by an increasing demand for evolved cloud services characterised by heterogeneous performance requirements including real-time, data-intensive and highly dynamic workloads. The classical way to deal with dynamicity is to scale computing and network resources horizontally. However, these techniques must be coupled effectively with advanced routing and switching in a multi-path environment, mixed with a high degree of flexibility to support dynamic adaptation and live-migration of virtual machines (VMs). We propose a management strategy to jointly optimise computing and networking resources in cloud infrastructures, where Software Defined Networking (SDN) plays a key enabling role