xDGP: A Dynamic Graph Processing System with Adaptive Partitioning

13 pagesMany real-world systems, such as social networks, rely on mining efficiently large graphs, with hundreds of millions of vertices and edges. This volume of information requires partitioning the graph across multiple nodes in a distributed system. This has a deep effect on performance, as traversing edges cut between partitions incurs a significant performance penalty due to the cost of communication. Thus, several systems in the literature have attempted to improve computational performance by enhancing graph partitioning, but they do not support another characteristic of real-world graphs: graphs are inherently dynamic, their topology evolves continuously, and subsequently the optimum partitioning also changes over time. In this work, we present the first system that dynamically repartitions massive graphs to adapt to structural changes. The system optimises graph partitioning to prevent performance degradation without using data replication. The system adopts an iterative vertex migration algorithm that relies on local information only, making complex coordination unnecessary. We show how the improvement in graph partitioning reduces execution time by over 50%, while adapting the partitioning to a large number of changes to the graph in three real-world scenarios

Robust Distributed Control Protocols for Large Vehicular Platoons with Prescribed Transient and Steady State Performance

In this paper, we study the longitudinal control problem for a platoon of vehicles with unknown nonlinear dynamics under both the predecessor-following and the bidirectional control architectures. The proposed control protocols are fully distributed in the sense that each vehicle utilizes feedback from its relative position with respect to its preceding and following vehicles as well as its own velocity, which can all be easily obtained by onboard sensors. Moreover, no previous knowledge of model nonlinearities/disturbances is incorporated in the control design, enhancing in that way the robustness of the overall closed loop system against model imperfections. Additionally, certain designer-specified performance functions determine the transient and steady-state response, thus preventing connectivity breaks due to sensor limitations as well as inter-vehicular collisions. Finally, extensive simulation studies and a real-time experiment conducted with mobile robots clarify the proposed control protocols and verify their effectiveness.Comment: IEEE Transactions on Control Systems Technology, accepte

Achieving high-throughput distributed, graph-based multi-stage stream processing

Includes bibliographical references.2015 Fall.Processing complex computations on high volume streaming data in real time is a challenge for many organizational data processing systems. Such systems should produce results with low latency while processing billions of messages daily. In order to address these requirements distributed stream processing systems have been developed. Although high performance is one of the main goals of these systems, there is less attention has been paid for inter node communication performance which is a key aspect to achieve overall system performance. In this thesis we describe a framework for enhancing inter node communication efficiency. We compare performance of our system with Twitter Storm and Yahoo S4 using an implementation of Pan Tompkins algorithm which is used to detect QRS complexities of an ECG signal using a 2 node graph. Our results show our solution performs 4 times better than other systems. We also use four level node graph which is used to process smart plug data to test the performance of our system for a complex graph. Finally we demonstrate how our system is scalable and resilient to faults

A Method for Securing Symmetric Keys for Internet of Things Enabled Distributed Data Systems

This study introduces an innovative method for securing symmetric keys in Internet of Things (IoT)-enabled distributed data systems, focusing on enhancing data security while optimizing encryption and decryption times. Through a comprehensive analysis of various encryption algorithms—TEA, XTEA, BLOCK TEA (XXTEA), and the proposed NTSA algorithm—across different key sizes and file sizes, we aim to demonstrate the significant improvements our method offers over existing techniques. Our research meticulously evaluated the performance of these algorithms, employing random variations to encryption and decryption times to simulate real-world variability and assess the algorithms' efficiency and security robustness. The findings reveal that the NTSA algorithm, in particular, showcases superior performance, offering an approximate improvement of 10% to 15% in encryption and decryption times over traditional methods such as TEA and XTEA, and an even more considerable enhancement compared to BLOCK TEA (XXTEA). The key contribution of this study lies in its provision of a secure, efficient framework for symmetric key encryption in IoT-enabled distributed environments. By optimizing key size and algorithm selection, our method not only secures data against potential cyber threats but also ensures high-speed data processing—a critical requirement in the IoT domain where the volume of data transactions and the need for real-time processing are ever-increasing. The proposed method significantly advances the field of data security in distributed systems, especially within the context of the burgeoning IoT landscape. It underscores the importance of algorithmic efficiency and strategic key management in bolstering the security and performance of modern digital ecosystems

A Symbiotic Approach to Designing Cross-Layer QoS in Embedded Real-Time Systems

International audienceNowadays there is an increasing need for embedded systems to support intensive computing while maintaining traditional hard real-time and fault-tolerant properties. Extending the principle of multi-core systems, we are exploring the use of distributed processing units interconnected via a high performance mesh network as a way of supporting distributed real-time applications. Fault-tolerance can then be ensured through dynamic allocation of both computing and communication resources. We postulate that enhancing QoS (Quality of Service) for real-time applications entails the development of a cross-layer support of high-level requirements, thus requiring a deep knowledge of the underlying networks. In this paper, we propose a new simulation/emulation/experimentation framework, ERICA, for designing such a feature. ERICA integrates both a network simulator and an actual hardware network to allow implementation and evaluation of different QoS-guaranteeing mechanisms. It also supports real-software-in-the-loop, i.e. running of real applications and middleware over these networks. Each component can evolve separately or together in a symbiotic manner, also making teamwork more flexible. We present in more detail our discrete-event simulation approach and the in-silicon implementation with which we cross-check our solutions in order to bring real performance aspects to our work. We also discuss the challenges of running real-software-in-the-loop in a real-time context, i.e. how to bridge it with a network simulator, and how to deal with time consistency

Enhancing the Spatio-Temporal Observability of Residential Loads

Enhancing the spatio-temporal observability of residential loads is crucial for achieving secure and efficient operations in distribution systems with increasing penetration of distributed energy resources (DERs). This paper presents a joint inference framework for residential loads by leveraging the real-time measurements from distribution-level sensors. Specifically, smart meter data is available for almost every load with unfortunately low temporal resolution, while distribution synchrophasor data is at very fast rates yet available at limited locations. By combining these two types of data with respective strengths, the problem is cast as a matrix recovery one with much less number of observations than unknowns. To improve the recovery performance, we introduce two regularization terms to promote a lowrank plus sparse structure of the load matrix via a difference transformation. Accordingly, the recovery problem can be formulated as a convex optimization one which is efficiently solvable. Numerical tests using real residential load data demonstrate the effectiveness of our proposed approaches in identifying appliance activities and recovering the PV output profiles

Optimal sizing of renewable energy storage: A comparative study of hydrogen and battery system considering degradation and seasonal storage

Renewable energy storage (RES) is essential to address the intermittence issues of renewable energy systems, thereby enhancing the system stability and reliability. This study presents an optimisation study of sizing and operational strategy parameters of a grid-connected photovoltaic (PV)-hydrogen/battery systems using a Multi-Objective Modified Firefly Algorithm (MOMFA). An operational strategy that utilises the ability of hydrogen to store energy over a long time was also investigated. The proposed method was applied to a real-world distributed energy project located in the tropical climate zone. To further demonstrate the robustness and versatility of the method, another synthetic test case was examined for a location in the subtropical weather zone, which has a high seasonal mismatch. The performance of the proposed MOMFA method is compared with the NSGA-II method, which has been widely used to design renewable energy storage systems in the literature. The result shows that MOMFA is more accurate and robust than NSGA-II owing to the complex and dynamic nature of energy storage system. The optimisation results show that battery storage systems, as a mature technology, yield better economic performance than current hydrogen storage systems. However, it is proven that hydrogen storage systems provide better techno-economic performance and can be a viable long-term storage solution when high penetration of renewable energy is required. The study also proves that the proposed long-term operational strategy can lower component degradation, enhance efficiency, and increase the total economic performance of hydrogen storage systems. The findings of this study can support the implementation of energy storage systems for renewable energy

Plant Information Management System Using OSIsoft PI System: A Case Study of Cilacap Power Plant Units 1 and 2

In the dynamic industrial sector, the ability to combine and manage data from various sources is essential. This study focuses on the utilization of the PI System to integrate operational data from multiple data sources at power plants, facilitating data driven decision making and enhancing operational efficiency. The application of the PI System in collecting data from Distributed Control Systems (DCS) and asset management systems like Maximo is examined, as well as its potential for integrating data from other power plant units. The study utilizes systems analysis methods to understand the potential of data integration in improving operational visibility and control. Findings from testing reveal that effective data integration can extend monitoring and management capabilities, indicating an overall improvement in operational performance. This study contributes to the literature on industrial data management by demonstrating the effective use of the PI System as a comprehensive tool for integrating data from various systems within power plants. It showcases the system's adaptability in enhancing operational decision-making and providing a cohesive platform for real-time performance monitoring. Additionally, the research offers insights into the practical application of systems analysis methods in the context of power plant operations, contributing to the ongoing discourse on digital transformation in the energy sector

A coordinated multi-element current differential protection scheme for active distribution systems

This paper introduces a current differential protection scheme, appropriate for application in medium voltage active distribution systems, where it is desired to keep the greatest possible number of loads and DG units energized during a fault. Conventional two-terminal percentage current differential relays are used to form successive, time-current-coordinated, differential protection zones. Multiple time-delayed differential elements in each protection zone guarantee coordination with the zone’s lateral protection devices, as well as between successive differential protection zones. Sensitive time-delayed differential elements protect against relatively high-resistance faults, while instantaneous differential elements minimize protection speed whenever possible. Additional emergency differential elements deal with post-fault topology changes and breaker failure conditions enhancing the overall scheme's performance. The proposed scheme is applied to a model of real medium voltage distribution system with distributed generation, considering a ring topology operation. A detailed simulation-based study proves the applicability and enhanced performance of the proposed scheme