A novel multi-level and community-based agent ecosystem to support customers dynamic decision-making in smart grids

Electrical systems have evolved at a fast pace over the past years, particularly in response to the current environmental and climate challenges. Consequently, the European Union and the United Nations have encouraged the development of a more sustainable energy strategy. This strategy triggered a paradigm shift in energy consumption and production, which becoming increasingly distributed, resulted in the development and emergence of smart energy grids. Multi-agent systems are one of the most widely used artificial intelligence concepts in smart grids. Both multi-agent systems and smart grids are distributed, so there is correspondence between the used technology and the network's complex reality. Due to the wide variety of multi-agent systems applied to smart grids, which typically have very specific goals, the ability to model the network as a whole may be compromised, as communication between systems is typically non-existent. This dissertation, therefore, proposes an agent-based ecosystem to model smart grids in which different agent-based systems can coexist. This dissertation aims to conceive, implement, test, and validate a new agent-based ecosystem, entitled A4SG (agent-based ecosystem for smart grids modelling), which combines the concepts of multi-agent systems and agent communities to enable the modelling and representation of smart grids and the entities that compose them. The proposed ecosystem employs an innovative methodology for managing static or dynamic interactions present in smart grids. The creation of a solution that allows the integration of existing systems into an ecosystem, enables the representation of smart grids in a realistic and comprehensive manner. A4SG integrates several functionalities that support the ecosystem's management, also conceived, implemented, tested, and validated in this dissertation. Two mobility functionalities are proposed: one that allows agents to move between physical machines and another that allows "virtual" mobility, where agents move between agent communities to improve the context for the achievement of their objectives. In order to prevent an agent from becoming overloaded, a novel functionality is proposed to enable the creation of agents that function as extensions of the main agent (i.e., branch agents), allowing the distribution of objectives among the various extensions of the main agent. Several case studies, which test the proposed services and functionalities individually and the ecosystem as a whole, were used to test and validate the proposed solution. These case studies were conducted in realistic contexts using data from multiple sources, including energy communities. The results indicate that the used methodologies can increase participation in demand response events, increasing the fitting between consumers and aggregators from 12 % to 69 %, and improve the strategies used in energy transaction markets, allowing an energy community of 50 customers to save 77.0 EUR per week.Os últimos anos têm sido de mudança nos sistemas elétricos, especialmente devido aos atuais desafios ambientais e climáticos. A procura por uma estratégia mais sustentável para o domínio da energia tem sido promovida pela União Europeia e pela Organização das Nações Unidas. A mudança de paradigma no que toca ao consumo e produção de energia, que acontece, cada vez mais, de forma distribuída, tem levado à emergência das redes elétricas inteligentes. Os sistemas multi-agente são um dos conceitos, no domínio da inteligência artificial, mais aplicados em redes inteligentes. Tanto os sistemas multi-agente como as redes inteligentes têm uma natureza distribuída, existindo por isso um alinhamento entre a tecnologia usada e a realidade complexa da rede. Devido a existir uma vasta oferta de sistemas multi-agente aplicados a redes inteligentes, normalmente com objetivos bastante específicos, a capacidade de modelar a rede como um todo pode ficar comprometida, porque a comunicação entre sistemas é, geralmente, inexistente. Por isso, esta dissertação propõe um ecossistema baseado em agentes para modelar as redes inteligentes, onde vários sistemas de agentes coexistem. Esta dissertação pretende conceber, implementar, testar, e validar um novo ecossistema multiagente, intitulado A4SG (agent-based ecosystem for smart grids modelling), que combina os conceitos de sistemas multi-agente e comunidades de agentes, permitindo a modelação e representação de redes inteligentes e das suas entidades. O ecossistema proposto utiliza uma metodologia inovadora para gerir as interações presentes nas redes inteligentes, sejam elas estáticas ou dinâmicas. A criação de um ecossistema que permite a integração de sistemas já existentes, cria a possibilidade de uma representação realista e detalhada das redes de energia. O A4SG integra diversas funcionalidades, também estas concebidas, implementadas, testadas, e validadas nesta dissertação, que suportam a gestão do próprio ecossistema. São propostas duas funcionalidades de mobilidade, uma que permite aos agentes mover-se entre máquinas físicas, e uma que permite uma mobilidade “virtual”, onde os agentes se movem entre comunidades de agentes, de forma a melhorar o contexto para a execução dos seus objetivos. É também proposta uma nova funcionalidade que permite a criação de agentes que funcionam como uma extensão de um agente principal, com o objetivo de evitar a sobrecarga de um agente, permitindo a distribuição de objetivos entre as várias extensões do agente principal. A solução proposta foi testada e validada por vários casos de estudo, que testam os serviços e funcionalidades propostas individualmente, e o ecossistema como um todo. Estes casos de estudo foram executados em contextos realistas, usando dados provenientes de diversas fontes, tais como comunidades de energia. Os resultados demonstram que as metodologias utilizadas podem melhorar a participação em eventos de demand response, subindo a adequação entre consumidores e agregadores de 12 % para 69 %, e melhorar as estratégias utilizadas em mercados de transações de energia, permitindo a uma comunidade de energia com 50 consumidores poupar 77,0 EUR por semana

Conflict resolution in mobile networks: a self-coordination framework based on non-dominated solutions and machine learning for data analytics [Application notes]

©2018 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.Self-organizing network (SON) is a well-known term used to describe an autonomous cellular network. SON functionalities aim at improving network operational tasks through the capability to configure, optimize and heal itself. However, as the deployment of independent SON functions increases, the number of dependencies between them also grows. This work proposes a tool for efficient conflict resolution based on network performance predictions. Unlike other state-of-theart solutions, the proposed self-coordination framework guarantees the right selection of network operation even if conflicting SON functions are running in parallel. This self-coordination is based on the history of network measurements, which helps to optimize conflicting objectives with low computational complexity. To do this, machine learning (ML) is used to build a predictive model, and then we solve the SON conflict by optimizing more than one objective function simultaneously. Without loss of generality, we present an analysis of how the proposed scheme provides a solution to deal with the potential conflicts between two of the most important SON functions in the context of mobility, namely mobility load balancing (MLB) and mobility robustness optimization (MRO), which require the updating of the same set of handover parameters. The proposed scheme allows fast performance evaluations when the optimization is running. This is done by shifting the complexity to the creation of a prediction model that uses historical data and that allows to anticipate the network performance. The simulation results demonstrate the ability of the proposed scheme to find a compromise among conflicting actions, and show it is possible to improve the overall system throughput.Peer ReviewedPostprint (author's final draft

Opinion Polarization by Learning from Social Feedback

We explore a new mechanism to explain polarization phenomena in opinion dynamics in which agents evaluate alternative views on the basis of the social feedback obtained on expressing them. High support of the favored opinion in the social environment, is treated as a positive feedback which reinforces the value associated to this opinion. In connected networks of sufficiently high modularity, different groups of agents can form strong convictions of competing opinions. Linking the social feedback process to standard equilibrium concepts we analytically characterize sufficient conditions for the stability of bi-polarization. While previous models have emphasized the polarization effects of deliberative argument-based communication, our model highlights an affective experience-based route to polarization, without assumptions about negative influence or bounded confidence.Comment: Presented at the Social Simulation Conference (Dublin 2017

Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges

Machine learning (ML) is widely used for key tasks in Connected and Automated Vehicles (CAV), including perception, planning, and control. However, its reliance on vehicular data for model training presents significant challenges related to in-vehicle user privacy and communication overhead generated by massive data volumes. Federated learning (FL) is a decentralized ML approach that enables multiple vehicles to collaboratively develop models, broadening learning from various driving environments, enhancing overall performance, and simultaneously securing local vehicle data privacy and security. This survey paper presents a review of the advancements made in the application of FL for CAV (FL4CAV). First, centralized and decentralized frameworks of FL are analyzed, highlighting their key characteristics and methodologies. Second, diverse data sources, models, and data security techniques relevant to FL in CAVs are reviewed, emphasizing their significance in ensuring privacy and confidentiality. Third, specific and important applications of FL are explored, providing insight into the base models and datasets employed for each application. Finally, existing challenges for FL4CAV are listed and potential directions for future work are discussed to further enhance the effectiveness and efficiency of FL in the context of CAV

A Theory of Mind Approach as Test-Time Mitigation Against Emergent Adversarial Communication

Multi-Agent Systems (MAS) is the study of multi-agent interactions in a shared environment. Communication for cooperation is a fundamental construct for sharing information in partially observable environments. Cooperative Multi-Agent Reinforcement Learning (CoMARL) is a learning framework where we learn agent policies either with cooperative mechanisms or policies that exhibit cooperative behavior. Explicitly, there are works on learning to communicate messages from CoMARL agents; however, non-cooperative agents, when capable of access a cooperative team's communication channel, have been shown to learn adversarial communication messages, sabotaging the cooperative team's performance particularly when objectives depend on finite resources. To address this issue, we propose a technique which leverages local formulations of Theory-of-Mind (ToM) to distinguish exhibited cooperative behavior from non-cooperative behavior before accepting messages from any agent. We demonstrate the efficacy and feasibility of the proposed technique in empirical evaluations in a centralized training, decentralized execution (CTDE) CoMARL benchmark. Furthermore, while we propose our explicit ToM defense for test-time, we emphasize that ToM is a construct for designing a cognitive defense rather than be the objective of the defense.Comment: 6 pages, 7 figure

Markov Decision Processes with Applications in Wireless Sensor Networks: A Survey

Wireless sensor networks (WSNs) consist of autonomous and resource-limited devices. The devices cooperate to monitor one or more physical phenomena within an area of interest. WSNs operate as stochastic systems because of randomness in the monitored environments. For long service time and low maintenance cost, WSNs require adaptive and robust methods to address data exchange, topology formulation, resource and power optimization, sensing coverage and object detection, and security challenges. In these problems, sensor nodes are to make optimized decisions from a set of accessible strategies to achieve design goals. This survey reviews numerous applications of the Markov decision process (MDP) framework, a powerful decision-making tool to develop adaptive algorithms and protocols for WSNs. Furthermore, various solution methods are discussed and compared to serve as a guide for using MDPs in WSNs

Social based adaptation in multi-agent systems

This work presents a new agent model based in three main topics: emotional psychology (OCC list of emotions), sociology and cognitive psychology (KAI index) and the concept of "opinion" given by the agents. The new model is suitable for implementing adaptation in changing environments, as well as for static ones which are highly unpredictable.García-Pardo Giménez De Los Galanes, JÁ. (2010). Social based adaptation in multi-agent systems. http://hdl.handle.net/10251/13702Archivo delegad

Benefits and limits of machine learning for the implicit coordination on SON functions

Bedingt durch die Einführung neuer Netzfunktionen in den Mobilfunknetzen der nächsten Generation, z. B. Slicing oder Mehrantennensysteme, sowie durch die Koexistenz mehrerer Funkzugangstechnologien, werden die Optimierungsaufgaben äußerst komplex und erhöhen die OPEX (OPerational EXpenditures). Um den Nutzern Dienste mit wettbewerbsfähiger Dienstgüte (QoS) zu bieten und gleichzeitig die Betriebskosten niedrig zu halten, wurde von den Standardisierungsgremien das Konzept des selbstorganisierenden Netzes (SON) eingeführt, um das Netzmanagement um eine Automatisierungsebene zu erweitern. Es wurden dafür mehrere SON-Funktionen (SFs) vorgeschlagen, um einen bestimmten Netzbereich, wie Abdeckung oder Kapazität, zu optimieren. Bei dem konventionellen Entwurf der SFs wurde jede Funktion als Regler mit geschlossenem Regelkreis konzipiert, der ein lokales Ziel durch die Einstellung bestimmter Netzwerkparameter optimiert. Die Beziehung zwischen mehreren SFs wurde dabei jedoch bis zu einem gewissen Grad vernachlässigt. Daher treten viele widersprüchliche Szenarien auf, wenn mehrere SFs in einem mobilen Netzwerk instanziiert werden. Solche widersprüchlichen Funktionen in den Netzen verschlechtern die QoS der Benutzer und beeinträchtigen die Signalisierungsressourcen im Netz. Es wird daher erwartet, dass eine existierende Koordinierungsschicht (die auch eine Entität im Netz sein könnte) die Konflikte zwischen SFs lösen kann. Da diese Funktionen jedoch eng miteinander verknüpft sind, ist es schwierig, ihre Interaktionen und Abhängigkeiten in einer abgeschlossenen Form zu modellieren. Daher wird maschinelles Lernen vorgeschlagen, um eine gemeinsame Optimierung eines globalen Leistungsindikators (Key Performance Indicator, KPI) so voranzubringen, dass die komplizierten Beziehungen zwischen den Funktionen verborgen bleiben. Wir nennen diesen Ansatz: implizite Koordination. Im ersten Teil dieser Arbeit schlagen wir eine zentralisierte, implizite und auf maschinellem Lernen basierende Koordination vor und wenden sie auf die Koordination zweier etablierter SFs an: Mobility Robustness Optimization (MRO) und Mobility Load Balancing (MLB). Anschließend gestalten wir die Lösung dateneffizienter (d. h. wir erreichen die gleiche Modellleistung mit weniger Trainingsdaten), indem wir eine geschlossene Modellierung einbetten, um einen Teil des optimalen Parametersatzes zu finden. Wir nennen dies einen "hybriden Ansatz". Mit dem hybriden Ansatz untersuchen wir den Konflikt zwischen MLB und Coverage and Capacity Optimization (CCO) Funktionen. Dann wenden wir ihn auf die Koordinierung zwischen MLB, Inter-Cell Interference Coordination (ICIC) und Energy Savings (ES) Funktionen an. Schließlich stellen wir eine Möglichkeit vor, MRO formal in den hybriden Ansatz einzubeziehen, und zeigen, wie der Rahmen erweitert werden kann, um anspruchsvolle Netzwerkszenarien wie Ultra-Reliable Low Latency Communications (URLLC) abzudecken.Due to the introduction of new network functionalities in next-generation mobile networks, e.g., slicing or multi-antenna systems, as well as the coexistence of multiple radio access technologies, the optimization tasks become extremely complex, increasing the OPEX (OPerational EXpenditures). In order to provide services to the users with competitive Quality of Service (QoS) while keeping low operational costs, the Self-Organizing Network (SON) concept was introduced by the standardization bodies to add an automation layer to the network management. Thus, multiple SON functions (SFs) were proposed to optimize a specific network domain, like coverage or capacity. The conventional design of SFs conceived each function as a closed-loop controller optimizing a local objective by tuning specific network parameters. However, the relationship among multiple SFs was neglected to some extent. Therefore, many conflicting scenarios appear when multiple SFs are instantiated in a mobile network. Having conflicting functions in the networks deteriorates the users’ QoS and affects the signaling resources in the network. Thus, it is expected to have a coordination layer (which could also be an entity in the network), conciliating the conflicts between SFs. Nevertheless, due to interleaved linkage among those functions, it is complex to model their interactions and dependencies in a closed form. Thus, machine learning is proposed to drive a joint optimization of a global Key Performance Indicator (KPI), hiding the intricate relationships between functions. We call this approach: implicit coordination. In the first part of this thesis, we propose a centralized, fully-implicit coordination approach based on machine learning (ML), and apply it to the coordination of two well-established SFs: Mobility Robustness Optimization (MRO) and Mobility Load Balancing (MLB). We find that this approach can be applied as long as the coordination problem is decomposed into three functional planes: controllable, environmental, and utility planes. However, the fully-implicit coordination comes at a high cost: it requires a large amount of data to train the ML models. To improve the data efficiency of our approach (i.e., achieving good model performance with less training data), we propose a hybrid approach, which mixes ML with closed-form models. With the hybrid approach, we study the conflict between MLB and Coverage and Capacity Optimization (CCO) functions. Then, we apply it to the coordination among MLB, Inter-Cell Interference Coordination (ICIC), and Energy Savings (ES) functions. With the hybrid approach, we find in one shot, part of the parameter set in an optimal manner, which makes it suitable for dynamic scenarios in which fast response is expected from a centralized coordinator. Finally, we present a manner to formally include MRO in the hybrid approach and show how the framework can be extended to cover challenging network scenarios like Ultra-Reliable Low Latency Communications (URLLC)

Towards a Better Understanding of Learning with Multiagent Teams

While it has long been recognized that a team of individual learning agents can be greater than the sum of its parts, recent work has shown that larger teams are not necessarily more effective than smaller ones. In this paper, we study why and under which conditions certain team structures promote effective learning for a population of individual learning agents. We show that, depending on the environment, some team structures help agents learn to specialize into specific roles, resulting in more favorable global results. However, large teams create credit assignment challenges that reduce coordination, leading to large teams performing poorly compared to smaller ones. We support our conclusions with both theoretical analysis and empirical results.Comment: 15 pages, 11 figures, published at the International Joint Conference on Artificial Intelligence (IJCAI) in 202