An Actor-Critic Reinforcement Learning Approach for Energy Harvesting Communications Systems

Energy harvesting communications systems are able to provide high quality communications services using green energy sources. This paper presents an autonomous energy harvesting communications system that is able to adapt to any environment, and optimize its behavior with experience to maximize the valuable received data. The considered system is a point-to-point energy harvesting communications system consisting of a source and a destination, and working in an unknown and uncertain environment. The source is an energy harvesting node capable of harvesting solar energy and storing it in a finite capacity battery. Energy can be harvested, stored, and used from continuous ranges of energy values. Channel gains can take any value within a continuous range. Since exact information about future channel gains and harvested energy is unavailable, an architecture based on actor-critic reinforcement learning is proposed to learn a close-to-optimal transmission power allocation policy. The actor uses a stochastic parameterized policy to select actions at states stochastically. The policy is modeled by a normal distribution with a parameterized mean and standard deviation. The actor uses policy gradient to optimize the policy’s parameters. The critic uses a three layer neural network to approximate the action-value function, and to evaluate the optimized policy. Simulation results evaluate the proposed architecture for actor-critic learning, and shows its ability to improve its performance with experience

Intelligent and secure fog-aided internet of drones

Internet of drones (IoD), which utilize drones as Internet of Things (IoT) devices, deploys several drones in the air to collect ground information and send them to the IoD gateway for further processing. Computing tasks are usually offloaded to the cloud data center for intensive processing. However, many IoD applications require real-time processing and event response (e.g., disaster response and virtual reality applications). Hence, data processing by the remote cloud may not satisfy the strict latency requirement. Fog computing attaches fog nodes, which are equipped with computing, storage and networking resources, to IoD gateways to assume a substantial amount of computing tasks instead of performing all tasks in the remote cloud, thus enabling immediate service response. Fog-aided IoD provisions future events prediction and image classification by machine learning technologies, where massive training data are collected by drones and analyzed in the fog node. However, the performance of IoD is greatly affected by drones\u27 battery capacities. Also, aggregating all data in the fog node may incur huge network traffic and drone data privacy leakage. To address the challenge of limited drone battery, the power control problem is first investigated in IoD for the data collection service to minimize the energy consumption of a drone while meeting the quality of service (QoS) requirements. A PowEr conTROL (PETROL) algorithm is then proposed to solve this problem and its convergence rate is derived. The task allocation (which distributes tasks to different fog nodes) and the flying control (which adjusts the drone\u27s flying speed) are then jointly optimized to minimize the drone\u27s journey completion time constrained by the drone\u27s battery capacity and task completion deadlines. In consideration of the practical scenario that the future task information is difficult to obtain, an online algorithm is designed to provide strategies for task allocation and flying control when the drone visits each location without knowing the future. The joint optimization of power control and energy harvesting control is also studied to determine each drone\u27s transmission power and the transmitted energy from the charging station in the time-varying IoD network. The objective is to minimize the long-term average system energy cost constrained by the drones\u27 battery capacities and QoS requirements. A Markov Decision Process (MDP) is formulated to characterize the power and energy harvesting control process in time-varying IoD networks. A modified actor-critic reinforcement learning algorithm is then proposed to tackle the problem. To address the challenge of drone data privacy leakage, federated learning (FL) is proposed to preserve drone data privacy by performing local training in drones and sharing training model parameters with a fog node without uploading drone raw data. However, drone privacy can still be divulged to ground eavesdroppers by wiretapping and analyzing uploaded parameters during the FL training process. The power control problem of all drones is hence investigated to maximize the FL system security rate constrained by drone battery capacities and the QoS requirements (e.g., FL training time). This problem is formulated as a non-linear programming problem and an algorithm is designed to obtain the optimum solutions with low computational complexity. All proposed algorithms are demonstrated to perform better than existing algorithms by extensive simulations and can be implemented in the intelligent and secure fog-aided IoD network to improve system performances on energy efficiency, QoS, and security