funcX: A Federated Function Serving Fabric for Science

Exploding data volumes and velocities, new computational methods and platforms, and ubiquitous connectivity demand new approaches to computation in the sciences. These new approaches must enable computation to be mobile, so that, for example, it can occur near data, be triggered by events (e.g., arrival of new data), be offloaded to specialized accelerators, or run remotely where resources are available. They also require new design approaches in which monolithic applications can be decomposed into smaller components, that may in turn be executed separately and on the most suitable resources. To address these needs we present funcX---a distributed function as a service (FaaS) platform that enables flexible, scalable, and high performance remote function execution. funcX's endpoint software can transform existing clouds, clusters, and supercomputers into function serving systems, while funcX's cloud-hosted service provides transparent, secure, and reliable function execution across a federated ecosystem of endpoints. We motivate the need for funcX with several scientific case studies, present our prototype design and implementation, show optimizations that deliver throughput in excess of 1 million functions per second, and demonstrate, via experiments on two supercomputers, that funcX can scale to more than more than 130000 concurrent workers.Comment: Accepted to ACM Symposium on High-Performance Parallel and Distributed Computing (HPDC 2020). arXiv admin note: substantial text overlap with arXiv:1908.0490

Secure Data Management and Transmission Infrastructure for the Future Smart Grid

Power grid has played a crucial role since its inception in the Industrial Age. It has evolved from a wide network supplying energy for incorporated multiple areas to the largest cyber-physical system. Its security and reliability are crucial to any country’s economy and stability [1]. With the emergence of the new technologies and the growing pressure of the global warming, the aging power grid can no longer meet the requirements of the modern industry, which leads to the proposal of ‘smart grid’. In smart grid, both electricity and control information communicate in a massively distributed power network. It is essential for smart grid to deliver real-time data by communication network. By using smart meter, AMI can measure energy consumption, monitor loads, collect data and forward information to collectors. Smart grid is an intelligent network consists of many technologies in not only power but also information, telecommunications and control. The most famous structure of smart grid is the three-layer structure. It divides smart grid into three different layers, each layer has its own duty. All these three layers work together, providing us a smart grid that monitor and optimize the operations of all functional units from power generation to all the end-customers [2]. To enhance the security level of future smart grid, deploying a high secure level data transmission scheme on critical nodes is an effective and practical approach. A critical node is a communication node in a cyber-physical network which can be developed to meet certain requirements. It also has firewalls and capability of intrusion detection, so it is useful for a time-critical network system, in other words, it is suitable for future smart grid. The deployment of such a scheme can be tricky regarding to different network topologies. A simple and general way is to install it on every node in the network, that is to say all nodes in this network are critical nodes, but this way takes time, energy and money. Obviously, it is not the best way to do so. Thus, we propose a multi-objective evolutionary algorithm for the searching of critical nodes. A new scheme should be proposed for smart grid. Also, an optimal planning in power grid for embedding large system can effectively ensure every power station and substation to operate safely and detect anomalies in time. Using such a new method is a reliable method to meet increasing security challenges. The evolutionary frame helps in getting optimum without calculating the gradient of the objective function. In the meanwhile, a means of decomposition is useful for exploring solutions evenly in decision space. Furthermore, constraints handling technologies can place critical nodes on optimal locations so as to enhance system security even with several constraints of limited resources and/or hardware. The high-quality experimental results have validated the efficiency and applicability of the proposed approach. It has good reason to believe that the new algorithm has a promising space over the real-world multi-objective optimization problems extracted from power grid security domain. In this thesis, a cloud-based information infrastructure is proposed to deal with the big data storage and computation problems for the future smart grid, some challenges and limitations are addressed, and a new secure data management and transmission strategy regarding increasing security challenges of future smart grid are given as well

Secure Data Management and Transmission Infrastructure for the Future Smart Grid

Power grid has played a crucial role since its inception in the Industrial Age. It has evolved from a wide network supplying energy for incorporated multiple areas to the largest cyber-physical system. Its security and reliability are crucial to any country’s economy and stability [1]. With the emergence of the new technologies and the growing pressure of the global warming, the aging power grid can no longer meet the requirements of the modern industry, which leads to the proposal of ‘smart grid’. In smart grid, both electricity and control information communicate in a massively distributed power network. It is essential for smart grid to deliver real-time data by communication network. By using smart meter, AMI can measure energy consumption, monitor loads, collect data and forward information to collectors. Smart grid is an intelligent network consists of many technologies in not only power but also information, telecommunications and control. The most famous structure of smart grid is the three-layer structure. It divides smart grid into three different layers, each layer has its own duty. All these three layers work together, providing us a smart grid that monitor and optimize the operations of all functional units from power generation to all the end-customers [2]. To enhance the security level of future smart grid, deploying a high secure level data transmission scheme on critical nodes is an effective and practical approach. A critical node is a communication node in a cyber-physical network which can be developed to meet certain requirements. It also has firewalls and capability of intrusion detection, so it is useful for a time-critical network system, in other words, it is suitable for future smart grid. The deployment of such a scheme can be tricky regarding to different network topologies. A simple and general way is to install it on every node in the network, that is to say all nodes in this network are critical nodes, but this way takes time, energy and money. Obviously, it is not the best way to do so. Thus, we propose a multi-objective evolutionary algorithm for the searching of critical nodes. A new scheme should be proposed for smart grid. Also, an optimal planning in power grid for embedding large system can effectively ensure every power station and substation to operate safely and detect anomalies in time. Using such a new method is a reliable method to meet increasing security challenges. The evolutionary frame helps in getting optimum without calculating the gradient of the objective function. In the meanwhile, a means of decomposition is useful for exploring solutions evenly in decision space. Furthermore, constraints handling technologies can place critical nodes on optimal locations so as to enhance system security even with several constraints of limited resources and/or hardware. The high-quality experimental results have validated the efficiency and applicability of the proposed approach. It has good reason to believe that the new algorithm has a promising space over the real-world multi-objective optimization problems extracted from power grid security domain. In this thesis, a cloud-based information infrastructure is proposed to deal with the big data storage and computation problems for the future smart grid, some challenges and limitations are addressed, and a new secure data management and transmission strategy regarding increasing security challenges of future smart grid are given as well

Resilience-driven planning and operation of networked microgrids featuring decentralisation and flexibility

High-impact and low-probability extreme events including both man-made events and natural weather events can cause severe damage to power systems. These events are typically rare but featured in long duration and large scale. Many research efforts have been conducted on the resilience enhancement of modern power systems. In recent years, microgrids (MGs) with distributed energy resources (DERs) including both conventional generation resources and renewable energy sources provide a viable solution for the resilience enhancement of such multi-energy systems during extreme events. More specifically, several islanded MGs after extreme events can be connected with each other as a cluster, which has the advantage of significantly reducing load shedding through energy sharing among them. On the other hand, mobile power sources (MPSs) such as mobile energy storage systems (MESSs), electric vehicles (EVs), and mobile emergency generators (MEGs) have been gradually deployed in current energy systems for resilience enhancement due to their significant advantages on mobility and flexibility. Given such a context, a literature review on resilience-driven planning and operation problems featuring MGs is presented in detail, while research limitations are summarised briefly. Then, this thesis investigates how to develop appropriate planning and operation models for the resilience enhancement of networked MGs via different types of DERs (e.g., MGs, ESSs, EVs, MESSs, etc.). This research is conducted in the following application scenarios: 1. This thesis proposes novel operation strategies for hybrid AC/DC MGs and networked MGs towards resilience enhancement. Three modelling approaches including centralised control, hierarchical control, and distributed control have been applied to formulate the proposed operation problems. A detailed non-linear AC OPF algorithm is employed to model each MG capturing all the network and technical constraints relating to stability properties (e.g., voltage limits, active and reactive power flow limits, and power losses), while uncertainties associated with renewable energy sources and load profiles are incorporated into the proposed models via stochastic programming. Impacts of limited generation resources, load distinction intro critical and non-critical, and severe contingencies (e.g., multiple line outages) are appropriately captured to mimic a realistic scenario. 2. This thesis introduces MPSs (e.g., EVs and MESSs) into the suggested networked MGs against the severe contingencies caused by extreme events. Specifically, time-coupled routing and scheduling characteristics of MPSs inside each MG are modelled to reduce load shedding when large damage is caused to each MG during extreme events. Both transportation networks and power networks are considered in the proposed models, while transporting time of MPSs between different transportation nodes is also appropriately captured. 3. This thesis focuses on developing realistic planning models for the optimal sizing problem of networked MGs capturing a trade-off between resilience and cost, while both internal uncertainties and external contingencies are considered in the suggested three-level planning model. Additionally, a resilience-driven planning model is developed to solve the coupled optimal sizing and pre-positioning problem of MESSs in the context of decentralised networked MGs. Internal uncertainties are captured in the model via stochastic programming, while external contingencies are included through the three-level structure. 4. This thesis investigates the application of artificial intelligence techniques to power system operations. Specifically, a model-free multi-agent reinforcement learning (MARL) approach is proposed for the coordinated routing and scheduling problem of multiple MESSs towards resilience enhancement. The parameterized double deep Q-network method (P-DDQN) is employed to capture a hybrid policy including both discrete and continuous actions. A coupled power-transportation network featuring a linearised AC OPF algorithm is realised as the environment, while uncertainties associated with renewable energy sources, load profiles, line outages, and traffic volumes are incorporated into the proposed data-driven approach through the learning procedure.Open Acces

Optimal and Secure Electricity Market Framework for Market Operation of Multi-Microgrid Systems

Traditional power systems were typically based on bulk energy services by large utility companies. However, microgrids and distributed generations have changed the structure of modern power systems as well as electricity markets. Therefore, restructured electricity markets are needed to address energy transactions in modern power systems. In this dissertation, we developed a hierarchical and decentralized electricity market framework for multi-microgrid systems, which clears energy transactions through three market levels; Day-Ahead-Market (DAM), Hour-Ahead-Market (HAM) and Real-Time-Market (RTM). In this market, energy trades are possible between all participants within the microgrids as well as inter-microgrids transactions. In this approach, we developed a game-theoretic-based double auction mechanism for energy transactions in the DAM, while HAM and RTM are cleared by an optimization algorithm and reverse action mechanism, respectively. For data exchange among market players, we developed a secure data-centric communication approach using the Data Distribution Service. Results demonstrated that this electricity market could significantly reduce the energy price and dependency of the multi-microgrid area on the external grid. Furthermore, we developed and verified a hierarchical blockchain-based energy transaction framework for a multi-microgrid system. This framework has a unique structure, which makes it possible to check the feasibility of energy transactions from the power system point of view by evaluating transmission system constraints. The blockchain ledger summarization, microgrid equivalent model development, and market players’ security and privacy enhancement are new approaches to this framework. The research in this dissertation also addresses some ancillary services in power markets such as an optimal power routing in unbalanced microgrids, where we developed a multi-objective optimization model and verified its ability to minimize the power imbalance factor, active power losses and voltage deviation in an unbalanced microgrid. Moreover, we developed an adaptive real-time congestion management algorithm to mitigate congestions in transmission systems using dynamic thermal ratings of transmission lines. Results indicated that the developed algorithm is cost-effective, fast, and reliable for real-time congestion management cases. Finally, we completed research about the communication framework and security algorithm for IEC 61850 Routable GOOSE messages and developed an advanced protection scheme as its application in modern power systems

Multi-community command and control systems in law enforcement: An introductory planning guide

A set of planning guidelines for multi-community command and control systems in law enforcement is presented. Essential characteristics and applications of these systems are outlined. Requirements analysis, system concept design, implementation planning, and performance and cost modeling are described and demonstrated with numerous examples. Program management techniques and joint powers agreements for multicommunity programs are discussed in detail. A description of a typical multi-community computer-aided dispatch system is appended

Training of Crisis Mappers and Map Production from Multi-sensor Data: Vernazza Case Study (Cinque Terre National Park, Italy)

This aim of paper is to presents the development of a multidisciplinary project carried out by the cooperation between Politecnico di Torino and ITHACA (Information Technology for Humanitarian Assistance, Cooperation and Action). The goal of the project was the training in geospatial data acquiring and processing for students attending Architecture and Engineering Courses, in order to start up a team of "volunteer mappers". Indeed, the project is aimed to document the environmental and built heritage subject to disaster; the purpose is to improve the capabilities of the actors involved in the activities connected in geospatial data collection, integration and sharing. The proposed area for testing the training activities is the Cinque Terre National Park, registered in the World Heritage List since 1997. The area was affected by flood on the 25th of October 2011. According to other international experiences, the group is expected to be active after emergencies in order to upgrade maps, using data acquired by typical geomatic methods and techniques such as terrestrial and aerial Lidar, close-range and aerial photogrammetry, topographic and GNSS instruments etc.; or by non conventional systems and instruments such us UAV, mobile mapping etc. The ultimate goal is to implement a WebGIS platform to share all the data collected with local authorities and the Civil Protectio

MARIAN: A hybrid, metric-driven, agent-based routing protocol for multihop ad-hoc networks

Recent advances in technology provided the ground for highly dynamic, mobile, infrastructure-less networks, namely, ad-hoc networks. Despite their enormous benefits, the full potential cannot be reached unless certain issues are resolved. These mainly involve routing, as the lack of an infrastructure imposes a heavy burden on mobile devices that must maintain location information and route data packets in a multi-hop fashion. Specifically, typical adhoc routing devices, such as Personal Digital Assistants (PDAs), are limited in respect to the available throughput, life-time, and performance, that these may provide, as routing elements. Thus, there is a need for metric-driven ad-hoc routing, that is, devices should be utilised for routing according to their fitness, as different device types significantly vary in terms of routing fitness. In addition, a concrete agent-based approach can provide a set of advantages over a non-agent-based one, which includes: better design practice; and automatic reconfigurability.This research work aims to investigate the applicability of stationary and mobile agent technology in multi-hop ad-hoc routing. Specifically, this research proposes a novel hybrid, metric-driven, agent-based routing protocol for multi-hop ad-hoc networks that will enhance current routing schemes. The novelties that are expected to be achieved include: maximum network performance, increased scalability, dynamic adaptation, Quality of Service (QoS), energy conservation, reconfigurability, and security. The underlying idea is based on the fact that stationary and mobile agents can be ideal candidates for such dynamic environments due to their advanced characteristics, and thus offer state of the art support in terms of organising the otherwise disoriented network into an efficient and flexible hierarchical structure, classifying the routing fitness of participating devices, and therefore allow intelligent routing decisions to be taken on that basis