Scientific Computing Meets Big Data Technology: An Astronomy Use Case

Scientific analyses commonly compose multiple single-process programs into a dataflow. An end-to-end dataflow of single-process programs is known as a many-task application. Typically, tools from the HPC software stack are used to parallelize these analyses. In this work, we investigate an alternate approach that uses Apache Spark -- a modern big data platform -- to parallelize many-task applications. We present Kira, a flexible and distributed astronomy image processing toolkit using Apache Spark. We then use the Kira toolkit to implement a Source Extractor application for astronomy images, called Kira SE. With Kira SE as the use case, we study the programming flexibility, dataflow richness, scheduling capacity and performance of Apache Spark running on the EC2 cloud. By exploiting data locality, Kira SE achieves a 2.5x speedup over an equivalent C program when analyzing a 1TB dataset using 512 cores on the Amazon EC2 cloud. Furthermore, we show that by leveraging software originally designed for big data infrastructure, Kira SE achieves competitive performance to the C implementation running on the NERSC Edison supercomputer. Our experience with Kira indicates that emerging Big Data platforms such as Apache Spark are a performant alternative for many-task scientific applications

Optimization approaches for exploiting the load flexibility of electric heating devices in smart grids

Energy systems all over the world are undergoing a fundamental transition to tackle climate change and other environmental challenges. The share of electricity generated by renewable energy sources has been steadily increasing. In order to cope with the intermittent nature of renewable energy sources, like photovoltaic systems and wind turbines, the electrical demand has to be adjusted to their power generation. To this end, flexible electrical loads are necessary. Moreover, optimization approaches and advanced information and communication technology can help to transform the traditional electricity grid into a smart grid. To shift the electricity consumption in time, electric heating devices, such as heat pumps or electric water heaters, provide significant flexibility. In order to exploit this flexibility, optimization approaches for controlling flexible devices are essential. Most studies in the literature use centralized optimization or uncoordinated decentralized optimization. Centralized optimization has crucial drawbacks regarding computational complexity, privacy, and robustness, but uncoordinated decentralized optimization leads to suboptimal results. In this thesis, coordinated decentralized and hybrid optimization approaches with low computational requirements are developed for exploiting the flexibility of electric heating devices. An essential feature of all developed methods is that they preserve the privacy of the residents. This cumulative thesis comprises four papers that introduce different types of optimization approaches. In Paper A, rule-based heuristic control algorithms for modulating electric heating devices are developed that minimize the heating costs of a residential area. Moreover, control algorithms for minimizing surplus energy that otherwise could be curtailed are introduced. They increase the self-consumption rate of locally generated electricity from photovoltaics. The heuristic control algorithms use a privacy-preserving control and communication architecture that combines centralized and decentralized control approaches. Compared to a conventional control strategy, the results of simulations show cost reductions of between 4.1% and 13.3% and reductions of between 38.3% and 52.6% regarding the surplus energy. Paper B introduces two novel coordinating decentralized optimization approaches for scheduling-based optimization. A comparison with different decentralized optimization approaches from the literature shows that the developed methods, on average, lead to 10% less surplus energy. Further, an optimization procedure is defined that generates a diverse solution pool for the problem of maximizing the self-consumption rate of locally generated renewable energy. This solution pool is needed for the coordination mechanisms of several decentralized optimization approaches. Combining the decentralized optimization approaches with the defined procedure to generate diverse solution pools, on average, leads to 100 kWh (16.5%) less surplus energy per day for a simulated residential area with 90 buildings. In Paper C, another decentralized optimization approach that aims to minimize surplus energy and reduce the peak load in a local grid is developed. Moreover, two methods that distribute a central wind power profile to the different buildings of a residential area are introduced. Compared to the approaches from the literature, the novel decentralized optimization approach leads to improvements of between 0.8% and 13.3% regarding the surplus energy and the peak load. Paper D introduces uncertainty handling control algorithms for modulating electricheating devices. The algorithms can help centralized and decentralized scheduling-based optimization approaches to react to erroneous predictions of demand and generation. The analysis shows that the developed methods avoid violations of the residents\u27 comfort limits and increase the self-consumption rate of electricity generated by photovoltaic systems. All introduced optimization approaches yield a good trade-off between runtime and the quality of the results. Further, they respect the privacy of residents, lead to better utilization of renewable energy, and stabilize the grid. Hence, the developed optimization approaches can help future energy systems to cope with the high share of intermittent renewable energy sources

Power Management of Nanogrid Cluster with P2P Electricity Trading Based on Future Trends of Load Demand and PV Power Production

This paper presents the power management of the nanogrid clusters assisted by a novel peer-to-peer(P2P) electricity trading. In our work, unbalance of power consumption among clusters is mitigated by the proposed P2P trading method. For power management of individual clusters, multi-objective optimization simultaneously minimizing total power consumption, portion of grid power consumption, and total delay incurred by scheduling is attempted. A renewable power source photovoltaic(PV) system is adopted for each cluster as a secondary source. The temporal surplus of self-supply PV power of a cluster can be sold through P2P trading to another cluster (s) experiencing temporal power shortage. The cluster in temporal shortage of electric power buys the PV power to reduce peak load and total delay. In P2P trading, a cooperative game model is used for buyers and sellers to maximize their welfare. To increase P2P trading efficiency, future trends of load demand and PV power production are considered for power management of each cluster to resolve instantaneous unbalance between load demand and PV power production. To this end, a gated recurrent unit network is used to forecast future load demand and future PV power production. Simulations verify the effectiveness of the proposed P2P trading for nanogrid clusters.Comment: This article is submitted for publication in Sustainable Cities and Societ

Demand response through decentralized optimization in residential areas with wind and photovoltaics

A paradigm shift has to be realized in future energy systems with high shares of renewable energy sources. The electrical demand has to react to the fluctuating electricity generation of renewable energy sources. To this end, flexible electrical loads like electric heating devices coupled with thermal storage or electric vehicles are necessary in combination with optimization approaches. In this paper, we develop a novel privacy-preserving approach for decentralized optimization to exploit load flexibility. This approach, which is based on a set of schedules, is referred to as SEPACO-IDA. The results show that our developed algorithm outperforms the other approaches for scheduling based decentralized optimization found in the literature. Furthermore, this paper clearly illustrates the suboptimal results for uncoordinated decentralized optimization and thus the strong need for coordination approaches. Another contribution of this paper is the development and evaluation of two methods for distributing a central wind power profile to the local optimization problem of distributed agents (Equal Distribution and Score-Rank-Proportional Distribution). These wind profile assignment methods are combined with different decentralized optimization approaches. The results reveal the dependency of the best wind profile assignment method on the used decentralized optimization approach

Bridging the Flexibility Concepts in the Buildings and Multi-energy Domains

paper aims to stimulate a discussion on how to create a bridge between the concept of flexibility used in power and energy systems and the flexibility that buildings can offer for providing services to the electrical system. The paper recalls the main concepts and approaches considered in the power systems and multi-energy systems, and summarises some aspects of flexibility in buildings. The overview shows that there is room to strengthen the contacts among the scientists operating in these fields. The common aim is to identify the complementary aspects and provide inputs to enhance the methodologies and models to enable and support an effective energy and ecologic transition

Cloudbus Toolkit for Market-Oriented Cloud Computing

This keynote paper: (1) presents the 21st century vision of computing and identifies various IT paradigms promising to deliver computing as a utility; (2) defines the architecture for creating market-oriented Clouds and computing atmosphere by leveraging technologies such as virtual machines; (3) provides thoughts on market-based resource management strategies that encompass both customer-driven service management and computational risk management to sustain SLA-oriented resource allocation; (4) presents the work carried out as part of our new Cloud Computing initiative, called Cloudbus: (i) Aneka, a Platform as a Service software system containing SDK (Software Development Kit) for construction of Cloud applications and deployment on private or public Clouds, in addition to supporting market-oriented resource management; (ii) internetworking of Clouds for dynamic creation of federated computing environments for scaling of elastic applications; (iii) creation of 3rd party Cloud brokering services for building content delivery networks and e-Science applications and their deployment on capabilities of IaaS providers such as Amazon along with Grid mashups; (iv) CloudSim supporting modelling and simulation of Clouds for performance studies; (v) Energy Efficient Resource Allocation Mechanisms and Techniques for creation and management of Green Clouds; and (vi) pathways for future research.Comment: 21 pages, 6 figures, 2 tables, Conference pape