An Emergent Architecture for Scaling Decentralized Communication Systems (DCS)

With recent technological advancements now accelerating the mobile and wireless Internet solution space, a ubiquitous computing Internet is well within the research and industrial community's design reach - a decentralized system design, which is not solely driven by static physical models and sound engineering principals, but more dynamically, perhaps sub-optimally at initial deployment and socially-influenced in its evolution. To complement today's Internet system, this thesis proposes a Decentralized Communication System (DCS) architecture with the following characteristics: flat physical topologies with numerous compute oriented and communication intensive nodes in the network with many of these nodes operating in multiple functional roles; self-organizing virtual structures formed through alternative mobility scenarios and capable of serving ad hoc networking formations; emergent operations and control with limited dependency on centralized control and management administration. Today, decentralized systems are not commercially scalable or viable for broad adoption in the same way we have to come to rely on the Internet or telephony systems. The premise in this thesis is that DCS can reach high levels of resilience, usefulness, scale that the industry has come to experience with traditional centralized systems by exploiting the following properties: (i.) network density and topological diversity; (ii.) self-organization and emergent attributes; (iii.) cooperative and dynamic infrastructure; and (iv.) node role diversity. This thesis delivers key contributions towards advancing the current state of the art in decentralized systems. First, we present the vision and a conceptual framework for DCS. Second, the thesis demonstrates that such a framework and concept architecture is feasible by prototyping a DCS platform that exhibits the above properties or minimally, demonstrates that these properties are feasible through prototyped network services. Third, this work expands on an alternative approach to network clustering using hierarchical virtual clusters (HVC) to facilitate self-organizing network structures. With increasing network complexity, decentralized systems can generally lead to unreliable and irregular service quality, especially given unpredictable node mobility and traffic dynamics. The HVC framework is an architectural strategy to address organizational disorder associated with traditional decentralized systems. The proposed HVC architecture along with the associated promotional methodology organizes distributed control and management services by leveraging alternative organizational models (e.g., peer-to-peer (P2P), centralized or tiered) in hierarchical and virtual fashion. Through simulation and analytical modeling, we demonstrate HVC efficiencies in DCS structural scalability and resilience by comparing static and dynamic HVC node configurations against traditional physical configurations based on P2P, centralized or tiered structures. Next, an emergent management architecture for DCS exploiting HVC for self-organization, introduces emergence as an operational approach to scaling DCS services for state management and policy control. In this thesis, emergence scales in hierarchical fashion using virtual clustering to create multiple tiers of local and global separation for aggregation, distribution and network control. Emergence is an architectural objective, which HVC introduces into the proposed self-management design for scaling and stability purposes. Since HVC expands the clustering model hierarchically and virtually, a clusterhead (CH) node, positioned as a proxy for a specific cluster or grouped DCS nodes, can also operate in a micro-capacity as a peer member of an organized cluster in a higher tier. As the HVC promotional process continues through the hierarchy, each tier of the hierarchy exhibits emergent behavior. With HVC as the self-organizing structural framework, a multi-tiered, emergent architecture enables the decentralized management strategy to improve scaling objectives that traditionally challenge decentralized systems. The HVC organizational concept and the emergence properties align with and the view of the human brain's neocortex layering structure of sensory storage, prediction and intelligence. It is the position in this thesis, that for DCS to scale and maintain broad stability, network control and management must strive towards an emergent or natural approach. While today's models for network control and management have proven to lack scalability and responsiveness based on pure centralized models, it is unlikely that singular organizational models can withstand the operational complexities associated with DCS. In this work, we integrate emergence and learning-based methods in a cooperative computing manner towards realizing DCS self-management. However, unlike many existing work in these areas which break down with increased network complexity and dynamics, the proposed HVC framework is utilized to offset these issues through effective separation, aggregation and asynchronous processing of both distributed state and policy. Using modeling techniques, we demonstrate that such architecture is feasible and can improve the operational robustness of DCS. The modeling emphasis focuses on demonstrating the operational advantages of an HVC-based organizational strategy for emergent management services (i.e., reachability, availability or performance). By integrating the two approaches, the DCS architecture forms a scalable system to address the challenges associated with traditional decentralized systems. The hypothesis is that the emergent management system architecture will improve the operational scaling properties of DCS-based applications and services. Additionally, we demonstrate structural flexibility of HVC as an underlying service infrastructure to build and deploy DCS applications and layered services. The modeling results demonstrate that an HVC-based emergent management and control system operationally outperforms traditional structural organizational models. In summary, this thesis brings together the above contributions towards delivering a scalable, decentralized system for Internet mobile computing and communications

Distributed reinforcement learning for self-reconfiguring modular robots

Thesis (Ph. D.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2007.This electronic version was submitted by the student author. The certified thesis is available in the Institute Archives and Special Collections.Includes bibliographical references (p. 101-106).In this thesis, we study distributed reinforcement learning in the context of automating the design of decentralized control for groups of cooperating, coupled robots. Specifically, we develop a framework and algorithms for automatically generating distributed controllers for self-reconfiguring modular robots using reinforcement learning. The promise of self-reconfiguring modular robots is that of robustness, adaptability and versatility. Yet most state-of-the-art distributed controllers are laboriously handcrafted and task-specific, due to the inherent complexities of distributed, local-only control. In this thesis, we propose and develop a framework for using reinforcement learning for automatic generation of such controllers. The approach is profitable because reinforcement learning methods search for good behaviors during the lifetime of the learning agent, and are therefore applicable to online adaptation as well as automatic controller design. However, we must overcome the challenges due to the fundamental partial observability inherent in a distributed system such as a self reconfiguring modular robot. We use a family of policy search methods that we adapt to our distributed problem. The outcome of a local search is always influenced by the search space dimensionality, its starting point, and the amount and quality of available exploration through experience.(cont) We undertake a systematic study of the effects that certain robot and task parameters, such as the number of modules, presence of exploration constraints, availability of nearest-neighbor communications, and partial behavioral knowledge from previous experience, have on the speed and reliability of learning through policy search in self-reconfiguring modular robots. In the process, we develop novel algorithmic variations and compact search space representations for learning in our domain, which we test experimentally on a number of tasks. This thesis is an empirical study of reinforcement learning in a simulated lattice based self-reconfiguring modular robot domain. However, our results contribute to the broader understanding of automatic generation of group control and design of distributed reinforcement learning algorithms.by Paulina Varshavskaya.Ph.D

Modelling, Monitoring, Control and Optimization for Complex Industrial Processes

This reprint includes 22 research papers and an editorial, collected from the Special Issue "Modelling, Monitoring, Control and Optimization for Complex Industrial Processes", highlighting recent research advances and emerging research directions in complex industrial processes. This reprint aims to promote the research field and benefit the readers from both academic communities and industrial sectors

Control and identification of non-linear systems using neural networks and reinforcement learning

Dissertação (mestrado)—Universidade de Brasília, Faculdade de Tecnologia, Departamento de Engenharia Elétrica, 2018.Este trabalho propõe um contolador adaptativo utilizando redes neuras e aprendizado por reforço para lidar com não-linearidades e variância no tempo. Para a realização de testes, um sistema de nível de líquidos de quarta ordem foi escolhido por apresentar uma gama de constantes de tempo e por possibilitar a mudança de parâmetros. O sistema foi identificado com redes neurais para prever estados futuros com o objetivo de compensar o atraso e melhorar a performance do controlador. Diversos testes foram realizados com diversas redes neurais para decidir qual rede neural seria utilizada para cada tarefa pertinente ao controlador. Os parâmetros do controlador foram ajustados e testados para que o controlador pudesse alcançar parâmetros arbitrários de performance. O controlador foi testado e comparado com o PI tradicional para validação e mostrou caracteristicas adaptativas e melhoria de performance ao longo do tempo, além disso, o controlador desenvolvido não necessita de informação prévia do sistema.Fundação de Apoio a Pesquisa do Distrito Federal (FAP-DF).This work presents a proposal of an adaptive controller using reinforcement learning and neural networks in order to deal with non-linearities and time-variance. To test the controller a fourth-order fluid level system was chosen because of its great range of time constants and the possibility of varying the system parameters. System identification was performed to predict future states of the system, bypass delay and enhance the controller’s performance. Several tests with different neural networks were made in order to decide which network would be assigned to which task. Various parameters of the controller were tested and tuned to achieve a controller that satisfied arbitrary specifications. The controller was tested against a conventional PI controller used as reference and has shown adaptive features and improvement during execution. Also, the proposed controller needs no previous information on the system in order to be designed

Network Synchronization and Control Based on Inverse Optimality : A Study of Inverter-Based Power Generation

This thesis dwells upon the synthesis of system-theoretical tools to understand and control the behavior of nonlinear networked systems. This work is at the crossroads of three topics: synchronization in coupled high-order oscillators, inverse optimal control and the application of inverter-based power systems. The control and stability of power systems leverages the theoretical results obtained for synchronization in coupled high-order oscillators and inverse optimal control.First, we study the dynamics of coupled high-order nonlinear oscillators. These are characterized by their rotational invariance, meaning that their dynamics remain unchanged following a static shift of their angles. We provide sufficient conditions for local frequency synchronization based on both direct, indirect Lyapunov methods and center manifold theory. Second, we study inverse optimal control problems, embedded in networked settings. In this framework, we depart from a given stabilizing control law, with an associated control Lyapunov function and reverse engineer the cost functional to guarantee the optimality of the controller. In this way, inverse optimal control generates a whole family of optimal controllers corresponding to different cost functions. This provides analytically explicit and numerically feasible solutions in closed-form. This approach circumvents the complexity of solving partial differential equations descending from dynamic programming and Bellman's principle of optimality. We show this to be the case also in the presence of disturbances in the dynamics and the cost. In networks, the controller obtained from inverse optimal control has a topological structure (e.g., it is distributed) and thus feasible for implementation. The tuning is analogous to that of linear quadratic regulators.Third, motivated by the pressing changes witnessed by the electrical grid toward renewable energy generation, we consider power system stability and control as the main application of this thesis. In particular, we apply our theoretical findings to study a network of power electronic inverters. We first propose a controller we term the matching controller, a control strategy that, based on DC voltage measurements, endows the inverters with an oscillatory behavior at a common desired frequency. In closed-loop with the matching control, inverters can be considered as nonlinear oscillators. Our study of the dynamics of nonlinear oscillator network provides feasible physical conditions that ask for damping on DC- and AC-side of each converter, that are sufficient for system-wide frequency synchronization.Furthermore, we showcase the usefulness of inverse optimal control for inverter-based generation at two different settings to synthesize robust angle controllers with respect to common disturbances in the grid and provable stability guarantees. All the controllers proposed in this thesis, provide the electrical grid with important services, namely power support whenever needed, as well as power sharing among all inverters

Deep Learning -Powered Computational Intelligence for Cyber-Attacks Detection and Mitigation in 5G-Enabled Electric Vehicle Charging Station

An electric vehicle charging station (EVCS) infrastructure is the backbone of transportation electrification. However, the EVCS has various cyber-attack vulnerabilities in software, hardware, supply chain, and incumbent legacy technologies such as network, communication, and control. Therefore, proactively monitoring, detecting, and defending against these attacks is very important. The state-of-the-art approaches are not agile and intelligent enough to detect, mitigate, and defend against various cyber-physical attacks in the EVCS system. To overcome these limitations, this dissertation primarily designs, develops, implements, and tests the data-driven deep learning-powered computational intelligence to detect and mitigate cyber-physical attacks at the network and physical layers of 5G-enabled EVCS infrastructure. Also, the 5G slicing application to ensure the security and service level agreement (SLA) in the EVCS ecosystem has been studied. Various cyber-attacks such as distributed denial of services (DDoS), False data injection (FDI), advanced persistent threats (APT), and ransomware attacks on the network in a standalone 5G-enabled EVCS environment have been considered. Mathematical models for the mentioned cyber-attacks have been developed. The impact of cyber-attacks on the EVCS operation has been analyzed. Various deep learning-powered intrusion detection systems have been proposed to detect attacks using local electrical and network fingerprints. Furthermore, a novel detection framework has been designed and developed to deal with ransomware threats in high-speed, high-dimensional, multimodal data and assets from eccentric stakeholders of the connected automated vehicle (CAV) ecosystem. To mitigate the adverse effects of cyber-attacks on EVCS controllers, novel data-driven digital clones based on Twin Delayed Deep Deterministic Policy Gradient (TD3) Deep Reinforcement Learning (DRL) has been developed. Also, various Bruteforce, Controller clones-based methods have been devised and tested to aid the defense and mitigation of the impact of the attacks of the EVCS operation. The performance of the proposed mitigation method has been compared with that of a benchmark Deep Deterministic Policy Gradient (DDPG)-based digital clones approach. Simulation results obtained from the Python, Matlab/Simulink, and NetSim software demonstrate that the cyber-attacks are disruptive and detrimental to the operation of EVCS. The proposed detection and mitigation methods are effective and perform better than the conventional and benchmark techniques for the 5G-enabled EVCS