Thirty Years of Machine Learning: The Road to Pareto-Optimal Wireless Networks

Future wireless networks have a substantial potential in terms of supporting a broad range of complex compelling applications both in military and civilian fields, where the users are able to enjoy high-rate, low-latency, low-cost and reliable information services. Achieving this ambitious goal requires new radio techniques for adaptive learning and intelligent decision making because of the complex heterogeneous nature of the network structures and wireless services. Machine learning (ML) algorithms have great success in supporting big data analytics, efficient parameter estimation and interactive decision making. Hence, in this article, we review the thirty-year history of ML by elaborating on supervised learning, unsupervised learning, reinforcement learning and deep learning. Furthermore, we investigate their employment in the compelling applications of wireless networks, including heterogeneous networks (HetNets), cognitive radios (CR), Internet of things (IoT), machine to machine networks (M2M), and so on. This article aims for assisting the readers in clarifying the motivation and methodology of the various ML algorithms, so as to invoke them for hitherto unexplored services as well as scenarios of future wireless networks.Comment: 46 pages, 22 fig

Multiagent-Based Control for Plug-and-Play Batteries in DC Microgrids with Infrastructure Compensation

The influence of the DC infrastructure on the control of power-storage flow in micro- and smart grids has gained attention recently, particularly in dynamic vehicle-to-grid charging applications. Principal effects include the potential loss of the charge–discharge synchronization and the subsequent impact on the control stabilization, the increased degradation in batteries’ health/life, and resultant power- and energy-efficiency losses. This paper proposes and tests a candidate solution to compensate for the infrastructure effects in a DC microgrid with a varying number of heterogeneous battery storage systems in the context of a multiagent neighbor-to-neighbor control scheme. Specifically, the scheme regulates the balance of the batteries’ load-demand participation, with adaptive compensation for unknown and/or time-varying DC infrastructure influences. Simulation and hardware-in-the-loop studies in realistic conditions demonstrate the improved precision of the charge–discharge synchronization and the enhanced balance of the output voltage under 24 h excessively continuous variations in the load demand. In addition, immediate real-time compensation for the DC infrastructure influence can be attained with no need for initial estimates of key unknown parameters. The results provide both the validation and verification of the proposals under real operational conditions and expectations, including the dynamic switching of the heterogeneous batteries’ connection (plug-and-play) and the variable infrastructure influences of different dynamically switched branches. Key observed metrics include an average reduced convergence time (0.66–13.366%), enhanced output-voltage balance (2.637–3.24%), power-consumption reduction (3.569–4.93%), and power-flow-balance enhancement (2.755–6.468%), which can be achieved for the proposed scheme over a baseline for the experiments in question.</p

Policy-based power consumption management in smart energy community using single agent and multi agent Q learning algorithms

Power consumption in residential sector has increased due to growing population, economic growth, invention of many electrical appliances and therefore is becoming a growing concern in the power industry. Managing power consumption in residential sector without sacrificing user comfort has become one of the main research areas recently. The complexity of the power system keeps growing due to the penetration of alternative sources of electric energy such as solar plant, Hydro, Biomass, Geothermal and wind farm to meet the growing demand for electricity. To overcome the challenges due to complexity, the power grid needs to be intelligent in all aspects. As the grid gets smarter and smarter, considerable efforts are being undertaken to make the houses and businesses smarter in consuming the electrical energy to minimize and level the electricity demand which is also known as Demand Side Management (DSM). It also necessitates that the conventional way of modelling, control and energy management in all sectors needs to be enhanced or replaced by intelligent information processing techniques. In our research work, it has been done in several stages. (Purpose of Study and Results) We proposed a policy-based framework which allows intelligent and flexible energy management of home appliances in a smart home which is complex and dynamic in ways that saves energy automatically. We considered the challenges in formalizing the behaviour of the appliances using their states and managing the energy consumption using policies. Policies are rules which are created and edited by a house agent to deal with situations or power problems that are likely to occur. Each time the power problem arises the house agent will refer to policy and one or a set of rules will be executed to overcome that situation. Our policy-based smart home can manage energy efficiently and can significantly participate in reducing peak energy demand (thereby may reduce carbon emission). Our proposed policy-based framework achieves peak shaving so that power consumption adapts to available power, while ensuring the comfort level of the inhabitants and taking device characteristics in to account. Our simulation results on MATLAB indicate that the proposed Policy driven homes can effectively contribute to Demand side power management by decreasing the peak hour usage of the appliances and can efficiently manage energy in a smart home in a user-friendly way. We propounded and developed peak demand management algorithms for a Smart Energy Community using different types of coordination mechanisms for coordination of multiple house agents working in the same environment. These algorithms use centralized model, decentralized model, hybrid model and Pareto resource allocation model for resource allocation. We modelled user comfort for the appliance based on user preference, the power reduction capability and the important activities that run around the house associated with that appliance. Moreover, we compared these algorithms with respect to their peak reduction capability, overall comfort of the community, simplicity of the algorithm and community involvement and finally able to find the best performing algorithm among them. Our simulation results show that the proposed coordination algorithms can effectively reduce peak demand while maintaining user comfort. With the help of our proposed algorithms, the demand for electricity of a smart community can be managed intelligently and sustainably. This work is not only aiming for peak reduction management it aims for achieving it while keeping the comfort level of the inhabitants is minimum. It can learn user’s behaviour and establish the set of optimal rules dynamically. If the available power to a house is kept at a certain level the house agent will learn to use this notional power to operate all the appliances according to the requirements and comfort level of the household. This way the consumers are forced to use the power below the set level which can result in the over-all power consumption be maintained at a certain rate or level which means sustainability is possible or depletion of natural resources for electricity can be reduced. Temporal interactions of Energy Demand by local users and renewable energy sources can also be done more efficiently by having a set of new policy rules to switch between the utility and the renewable source of energy but it is beyond the scope of this thesis. We applied Q learning techniques to a home energy management agent where the agent learns to find the optimal sequence of turning off appliances so that the appliances with higher priority will not be switched off during peak demand period or power consumption management. The policy-based home energy management determines the optimal policy at every instant dynamically by learning through the interaction with the environment using one of the reinforcement learning approaches called Q-learning. The Q-learning home power consumption problem formulation consisting of state space, actions and reward function is presented. The implications of these simulation results are that the proposed Q- learning based power consumption management is very effective and enables the users to have minimum discomfort during participation in peak demand management or at the time when power consumption management is essential when the available power is rationale. This work is extended to a group of 10 houses and three multi agent Q- learning algorithms are proposed and developed for improving the individual and community comfort while at the same time keeping the power consumption below the available power level or electricity price below the set price. The proposed algorithms are weighted strategy sharing algorithm, concurrent Q learning algorithm and cooperative distributive learning algorithm. These proposed algorithms are coded and tested for managing power consumption of a group of 10 houses and the performance of all three algorithms with respect to power management and community comfort is studied and compared. Actual power consumption of a community and modified power consumption curves using Weighted Strategy Sharing algorithm, Concurrent learning and Distributive Q Learning and user comfort results are presented, and the results are analysed in this thesis

Reinforcement Learning and Game Theory for Smart Grid Security

This dissertation focuses on one of the most critical and complicated challenges facing electric power transmission and distribution systems which is their vulnerability against failure and attacks. Large scale power outages in Australia (2016), Ukraine (2015), India (2013), Nigeria (2018), and the United States (2011, 2003) have demonstrated the vulnerability of power grids to cyber and physical attacks and failures. These incidents clearly indicate the necessity of extensive research efforts to protect the power system from external intrusion and to reduce the damages from post-attack effects. We analyze the vulnerability of smart power grids to cyber and physical attacks and failures, design different gametheoretic approaches to identify the critical components vulnerable to attack and propose their associated defense strategy, and utilizes machine learning techniques to solve the game-theoretic problems in adversarial and collaborative adversarial power grid environment. Our contributions can be divided into three major parts:Vulnerability identification: Power grid outages have disastrous impacts on almost every aspect of modern life. Despite their inevitability, the effects of failures on power grids’ performance can be limited if the system operator can predict and identify the vulnerable elements of power grids. To enable these capabilities we study machine learning algorithms to identify critical power system elements adopting a cascaded failure simulator as a threat and attack model. We use generation loss, time to reach a certain percentage of line outage/generation loss, number of line outages, etc. as evaluation metrics to evaluate the consequences of threat and attacks on the smart power grid.Adversarial gaming in power system: With the advancement of the technologies, the smart attackers are deploying different techniques to supersede the existing protection scheme. In order to defend the power grid from these smart attackers, we introduce an adversarial gaming environment using machine learning techniques which is capable of replicating the complex interaction between the attacker and the power system operators. The numerical results show that a learned defender successfully narrows down the attackers’ attack window and reduce damages. The results also show that considering some crucial factors, the players can independently execute actions without detailed information about each other.Deep learning for adversarial gaming: The learning and gaming techniques to identify vulnerable components in the power grid become computationally expensive for large scale power systems. The power system operator needs to have the advanced skills to deal with the large dimensionality of the problem. In order to aid the power system operator in finding and analyzing vulnerability for large scale power systems, we study a deep learning technique for adversary game which is capable of dealing with high dimensional power system state space with less computational time and increased computational efficiency. Overall, the results provided in this dissertation advance power grids’ resilience and security by providing a better understanding of the systems’ vulnerability and by developing efficient algorithms to identify vulnerable components and appropriate defensive strategies to reduce the damages of the attack

Five Facets of 6G: Research Challenges and Opportunities

Whilst the fifth-generation (5G) systems are being rolled out across the globe, researchers have turned their attention to the exploration of radical next-generation solutions. At this early evolutionary stage we survey five main research facets of this field, namely {\em Facet~1: next-generation architectures, spectrum and services, Facet~2: next-generation networking, Facet~3: Internet of Things (IoT), Facet~4: wireless positioning and sensing, as well as Facet~5: applications of deep learning in 6G networks.} In this paper, we have provided a critical appraisal of the literature of promising techniques ranging from the associated architectures, networking, applications as well as designs. We have portrayed a plethora of heterogeneous architectures relying on cooperative hybrid networks supported by diverse access and transmission mechanisms. The vulnerabilities of these techniques are also addressed and carefully considered for highlighting the most of promising future research directions. Additionally, we have listed a rich suite of learning-driven optimization techniques. We conclude by observing the evolutionary paradigm-shift that has taken place from pure single-component bandwidth-efficiency, power-efficiency or delay-optimization towards multi-component designs, as exemplified by the twin-component ultra-reliable low-latency mode of the 5G system. We advocate a further evolutionary step towards multi-component Pareto optimization, which requires the exploration of the entire Pareto front of all optiomal solutions, where none of the components of the objective function may be improved without degrading at least one of the other components

Reinforcement Learning and Bandits for Speech and Language Processing: Tutorial, Review and Outlook

In recent years, reinforcement learning and bandits have transformed a wide range of real-world applications including healthcare, finance, recommendation systems, robotics, and last but not least, the speech and natural language processing. While most speech and language applications of reinforcement learning algorithms are centered around improving the training of deep neural networks with its flexible optimization properties, there are still many grounds to explore to utilize the benefits of reinforcement learning, such as its reward-driven adaptability, state representations, temporal structures and generalizability. In this survey, we present an overview of recent advancements of reinforcement learning and bandits, and discuss how they can be effectively employed to solve speech and natural language processing problems with models that are adaptive, interactive and scalable.Comment: To appear in Expert Systems with Applications. Accompanying INTERSPEECH 2022 Tutorial on the same topic. Including latest advancements in large language models (LLMs

Developing A Multi-Agent and Self-Adaptive Framework with Deep Reinforcement Learning for Dynamic Portfolio Risk Management

Deep or reinforcement learning (RL) approaches have been adapted as reactive agents to quickly learn and respond with new investment strategies for portfolio management under the highly turbulent financial market environments in recent years. In many cases, due to the very complex correlations among various financial sectors, and the fluctuating trends in different financial markets, a deep or reinforcement learning based agent can be biased in maximising the total returns of the newly formulated investment portfolio while neglecting its potential risks under the turmoil of various market conditions in the global or regional sectors. Accordingly, a multi-agent and self-adaptive framework namely the MASA is proposed in which a sophisticated multi-agent reinforcement learning (RL) approach is adopted through two cooperating and reactive agents to carefully and dynamically balance the trade-off between the overall portfolio returns and their potential risks. Besides, a very flexible and proactive agent as the market observer is integrated into the MASA framework to provide some additional information on the estimated market trends as valuable feedbacks for multi-agent RL approach to quickly adapt to the ever-changing market conditions. The obtained empirical results clearly reveal the potential strengths of our proposed MASA framework based on the multi-agent RL approach against many well-known RL-based approaches on the challenging data sets of the CSI 300, Dow Jones Industrial Average and S&P 500 indexes over the past 10 years. More importantly, our proposed MASA framework shed lights on many possible directions for future investigation.Comment: Accepted by The 23rd International Conference on Autonomous Agents and Multi-Agent System