A Holistic Approach to Forecasting Wholesale Energy Market Prices

Electricity market price predictions enable energy market participants to shape their consumption or supply while meeting their economic and environmental objectives. By utilizing the basic properties of the supply-demand matching process performed by grid operators, known as Optimal Power Flow (OPF), we develop a methodology to recover energy market's structure and predict the resulting nodal prices by using only publicly available data, specifically grid-wide generation type mix, system load, and historical prices. Our methodology uses the latest advancements in statistical learning to cope with high dimensional and sparse real power grid topologies, as well as scarce, public market data, while exploiting structural characteristics of the underlying OPF mechanism. Rigorous validations using the Southwest Power Pool (SPP) market data reveal a strong correlation between the grid level mix and corresponding market prices, resulting in accurate day-ahead predictions of real time prices. The proposed approach demonstrates remarkable proximity to the state-of-the-art industry benchmark while assuming a fully decentralized, market-participant perspective. Finally, we recognize the limitations of the proposed and other evaluated methodologies in predicting large price spike values.Comment: 14 pages, 14 figures. Accepted for publication in IEEE Transactions on Power System

Pipe failure prediction and impacts assessment in a water distribution network

Abstract Water distribution networks (WDNs) aim to provide water with desirable quantity, quality and pressure to the consumers. However, in case of pipe failure, which is the cumulative effect of physical, operational and weather-related factors, the WDN might fail to meet these objectives. Rehabilitation and replacement of some components of WDNs, such as pipes, is a common practice to improve the condition of the network to provide an acceptable level of service. The overall aim of this thesis is to predict—long-term, annually and short-term—the pipe failure propensity and assess the impacts of a single pipe failure on the level of service. The long-term and annual predictions facilitate the need for effective capital investment, whereas the short-term predictions have an operational use, enabling the water utilities to adjust the daily allocation and planning of resources to accommodate possible increase in pipe failure. The proposed methodology was implemented to the cast iron (CI) pipes in a UK WDN. The long-term and annual predictions are made using a novel combination of Evolutionary Polynomial Regression (EPR) and K-means clustering. The inclusion of K-means improves the predictions’ accuracy by using a set of models instead of a single model. The long-term predictive models consider physical factors, while the annual predictions also include weather-related factors. The analysis is conducted on a group level assuming that pipes with similar properties have similar breakage patterns. Soil type is another aggregation criterion since soil properties are associated with the corrosion of metallic pipes. The short-term predictions are based on a novel Artificial Neural Network (ANN) model that predicts the variations above a predefined threshold in the number of failures in the following days. The ANN model uses only existing weather data to make predictions reducing their uncertainty. The cross-validation technique is used to derive an accurate estimate of accuracy of EPR and ANN models by guaranteeing that all observations are used for both training and testing, and each observation is used for testing only once. The impact of pipe failure is assessed considering its duration, the topology of the network, the geographic location of the failed pipe and the time. The performance indicators used are the ratio of unsupplied demand and the number of customers with partial or no supply. Two scenarios are examined assuming that the failure occurs when there is a peak in either pressure or demand. The pressure-deficient conditions are simulated by introducing a sequence of artificial elements to all the demand nodes with pressure less than the required. This thesis proposes a new combination of a group-based method for deriving the failure rate and an individual-pipe method for evaluating the impacts on the level of service. Their conjunction indicates the most critical pipes. The long-term approach improves the accuracy of predictions, particularly for the groups with very low or very high failure frequency, considering diameter, age and length. The annual predictions accurately predict the fluctuation of failure frequency and its peak during the examined period. The EPR models indicate a strong direct relationship between low temperatures and failure frequency. The short-term predictions interpret the intra-year variation of failure frequency, with most failures occurring during the coldest months. The exhaustive trials led to the conclusion that the use of four consecutive days as input and the following two days as output results in the highest accuracy. The analysis of the relative significance of each input variable indicates that the variables that capture the intensity of low temperatures are the most influential. The outputs of the impact assessment indicate that the failure of most of the pipes in both scenarios (i.e. peak in pressure and demand) would have low impacts (i.e. low ratio of unsupplied demand and small number of affected nodes). This can be explained by the fact that the examined network is a large real-life network, and a single failure of a distribution pipe is likely to cause pressure-deficient conditions in a small part of it, whereas performance elsewhere is mostly satisfactory. Furthermore, the complex structure of the WDN allows them to recover from local pipe failures, exploiting the topological redundancy provided by closed loops, so that the flow could reach a given demand node through alternative paths

A pipeline architecture for feature-based unsupervised clustering using multivariate time series from HPC jobs

[Abstract]: Time series are key across industrial and research areas for their ability to model behaviour across time, making them ideal for a wide range of use cases such as event monitoring, trend prediction or anomaly detection. This is even more so due to the increasing monitoring capabilities in many areas, with the subsequent massive data generation. But it is also interesting to consider the potential of time series for Machine Learning processing, often fused with Big Data, to search for useful information and solve real-world problems. However, time series can be studied individually, representing a single entity or variable to be analysed, or in a grouped fashion, to study and represent a more complex entity or scenario. In this latter case we are dealing with multivariate time series, which usually imply different approaches when dealt with. In this paper, we present a pipeline architecture to process and cluster multiple groups of multivariate time series. To implement this, we apply a multi-process solution composed by a feature-based extraction stage, followed by a dimension reduction, and finally, several clustering algorithms. The pipeline is also highly configurable in terms of the stage techniques to be used, allowing to perform a search with several combinations for the most promising results. The pipeline has been experimentally applied to batches of HPC jobs from different users of a supercomputer, with the multivariate time series coming from the monitoring of several node resource metrics. The results show how it is possible to apply this multi-process information fusion to create different meaningful clusters from the batches, using only the time series, without any labelling information, thus being an unsupervised scenario. Optionally, the pipeline also supports an outlier detection stage to find and separate jobs that are radically different when compared to others on a dataset. These outliers can be removed for a better clustering, and later reviewed looking for anomalies, or if numerous, fed back to the pipeline to identify possible groupings. The results also include some outliers found in the experiments, as well as scenarios where they are clustered, or ignored and not removed at all. In addition, by leveraging Big Data technologies like Spark, the pipeline is proven to be scalable by working with up to hundreds of jobs and thousands of time series.Xunta de Galicia; ED431G 2019/01Xunta de Galicia; ED431C 2021/30This research was funded by the Ministry of Science and Innovation of Spain (PID2019-104184RB-I00/AEI/10.13039/501100011033), and by Xunta de Galicia, Spain and FEDER funds of the European Union (Centro de Investigación de Galicia accreditation 2019–2022, ref. ED431G 2019/01; Consolidation Program of Competitive Reference Groups, ref. ED431C 2021/30). Funding for open access charge: Universidade da Coruña/CISUG

Classification of damage in structural systems using time series analysis and supervised and unsupervised pattern recognition techniques

Peer reviewedPostprin

Major challenges in prognostics: study on benchmarking prognostic datasets

Even though prognostics has been defined to be one of the most difficult tasks in Condition Based Maintenance (CBM), many studies have reported promising results in recent years. The nature of the prognostics problem is different from diagnostics with its own challenges. There exist two major approaches to prognostics: data-driven and physics-based models. This paper aims to present the major challenges in both of these approaches by examining a number of published datasets for their suitability for analysis. Data-driven methods require sufficient samples that were run until failure whereas physics-based methods need physics of failure progression