Evolutionary multivariate time series prediction

Multivariate time series (MTS) prediction plays a significant role in many practical data mining applications, such as finance, energy supply, and medical care domains. Over the years, various prediction models have been developed to obtain robust and accurate prediction. However, this is not an easy task by considering a variety of key challenges. First, not all channels (each channel represents one time series) are informative (channel selection). Considering the complexity of each selected time series, it is difficult to predefine a time window used for inputs. Second, since the selected time series may come from cross domains collected with different devices, they may require different feature extraction techniques by considering suitable parameters to extract meaningful features (feature extraction), which influences the selection and configuration of the predictor, i.e., prediction (configuration). The challenge arising from channel selection, feature extraction, and prediction (configuration) is to perform them jointly to improve prediction performance. Third, we resort to ensemble learning to solve the MTS prediction problem composed of the previously mentioned operations,  where the challenge is to obtain a set of models satisfied both accurate and diversity. Each of these challenges leads to an NP-hard combinatorial optimization problem, which is impossible to be solved using the traditional methods since it is non-differentiable. Evolutionary algorithm (EA), as an efficient metaheuristic stochastic search technique, which is highly competent to solve complex combinatorial optimization problems having mixed types of decision variables, may provide an effective way to address the challenges arising from MTS prediction. The main contributions are supported by the following investigations. First, we propose a discrete evolutionary model, which mainly focuses on seeking the influential subset of channels of MTS and the optimal time windows for each of the selected channels for the MTS prediction task. A comprehensively experimental study on a real-world electricity consumption data with auxiliary environmental factors demonstrates the efficiency and effectiveness of the proposed method in searching for the informative time series and respective time windows and parameters in a predictor in comparison to the result obtained through enumeration. Subsequently, we define the basic MTS prediction pipeline containing channel selection, feature extraction, and prediction (configuration). To perform these key operations, we propose an evolutionary model construction (EMC) framework to seek the optimal subset of channels of MTS, suitable feature extraction methods and respective time windows applied to the selected channels, and parameter settings in the predictor simultaneously for the best prediction performance. To implement EMC, a two-step EA is proposed, where the first step EA mainly focuses on channel selection while in the second step, a specially designed EA works on feature extraction and prediction (configuration). A real-world electricity data with exogenous environmental information is used and the whole dataset is split into another two datasets according to holiday and nonholiday events. The performance of EMC is demonstrated on all three datasets in comparison to hybrid models and some existing methods. Then, based on the prediction pipeline defined previously, we propose an evolutionary multi-objective ensemble learning model (EMOEL) by employing multi-objective evolutionary algorithm (MOEA) subjected to two conflicting objectives, i.e., accuracy and model diversity. MOEA leads to a pareto front (PF) composed of non-dominated optimal solutions, where each of them represents the optimal subset of the selected channels, the selected feature extraction methods and the selected time windows, and the selected parameters in the predictor. To boost ultimate prediction accuracy, the models with respect to these optimal solutions are linearly combined with combination coefficients being optimized via a single-objective task-oriented EA. The superiority of EMOEL is identified on electricity consumption data with climate information in comparison to several state-of-the-art models. We also propose a multi-resolution selective ensemble learning model, where multiple resolutions are constructed from the minimal granularity using statistics. At the current time stamp, the preceding time series data is sampled at different time intervals (i.e., resolutions) to constitute the time windows. For each resolution, multiple base learners with different parameters are first trained. Feature selection technique is applied to search for the optimal set of trained base learners and least square regression is used to combine them. The performance of the proposed ensemble model is verified on the electricity consumption data for the next-step and next-day prediction. Finally, based on EMOEL and multi-resolution, instead of only combining the models generated from each PF, we propose an evolutionary ensemble learning (EEL) framework, where multiple PFs are aggregated to produce a composite PF (CPF) after removing the same solutions in PFs and being sorted into different levels of non-dominated fronts (NDFs). Feature selection techniques are applied to exploit the optimal subset of models in level-accumulated NDF and least square is used to combine the selected models. The performance of EEL that chooses three different predictors as base learners is evaluated by the comprehensive analysis of the parameter sensitivity. The superiority of EEL is demonstrated in comparison to the best result from single-objective EA and the best individual from the PF, and several state-of-the-art models across electricity consumption and air quality datasets, both of which use the environmental factors from other domains as the auxiliary factors. In summary, this thesis provides studies on how to build efficient and effective models for MTS prediction. The built frameworks investigate the influential factors, consider the pipeline composed of channel selection, feature extraction, and prediction (configuration) simultaneously, and keep good generalization and accuracy across different applications. The proposed algorithms to implement the frameworks use techniques from evolutionary computation (single-objective EA and MOEA), machine learning and data mining areas. We believe that this research provides a significant step towards constructing robust and accurate models for solving MTS prediction problems. In addition, with the case study on electricity consumption prediction, it will contribute to helping decision-makers in determining the trend of future energy consumption for scheduling and planning of the operations of the energy supply system

Towards intelligent operation of future power system: bayesian deep learning based uncertainty modelling technique

The increasing penetration level of renewable energy resources (RES) in the power system brings fundamental changes of the system operating paradigms. In the future, the intermittent nature of RES and the corresponding smart grid technologies will lead to a much more volatile power system with higher level uncertainties. At the same time, as a result of the larger scale installation of advanced sensor devices in power system, power system engineers for the first time have the opportunity to gain insights from the influx of massive data sets in order to improve the system performance in various aspects. To this end, it is imperative to explore big data methodologies with the aim of exploring the uncertainty space within such complex data sets and thus supporting real-time decision-making in future power system. In this thesis, Bayesian Deep learning is investigated with the aim of exploring data-driven methodologies to deal with uncertainties which is in the following three aspects. (1) The first part of this thesis proposes a novel probabilistic day-ahead net load forecasting method to capture both epistemic uncertainty and aleatoric uncertainty using Bayesian deep long short-term memory network. The proposed methodological framework employs clustering in sub-profiles and considers residential rooftop PV outputs as input features to enhance the performance of aggregated net load forecasting. Numerical experiments have been carried out based on fine-grained smart meter data from the Australian grid with separately recorded measurements of rooftop PV generation and loads. The results demonstrate the superior performance of the proposed scheme compared with a series of state-of-the-art methods and indicate the importance and effectiveness of sub-profile clustering and high PV visibility. (2) The second part of this thesis studies a novel Conditional Bayesian Deep Auto-Encoder (CBDAC) based security assessment framework to compute a confidence metric of the prediction. This informs not only the operator to judge whether the prediction can be trusted, but it also allows for judging whether the model needs updating. A case study based on IEEE 68-bus system demonstrates that CBDAC outperforms the state-of-the-art machine learning-based DSA methods and the models that need updating under different topologies can be effectively identified. Furthermore, the case study verifies that effective updating of the models is possible even with very limited data. (3) The last part of this thesis proposes a novel Bayesian Deep Reinforcement Learning-based resilient control approach for multi-energy micro-grid. In particular, the proposed approach replaces deterministic network in traditional Reinforcement Learning with Bayesian probabilistic network in order to obtain an approximation of the value function distribution, which effectively solves Q-value overestimation issue. The proposed model is able to provide both energy management during normal operating conditions and resilient control during extreme events in a multi-energy micro-grid system. Comparing with naive DDPG method and optimisation method, the effectiveness and importance of employing Bayesian Reinforcement Learning approach is investigated and illustrated across different operating scenarios. Case studies have shown that by using the Monte Carlo posterior mean of the Bayesian value function distribution instead of a deterministic estimation, the proposed BDDPG method achieves a near-optimum policy in a more stable process, which verifies the robustness and the practicability of the proposed approach.Open Acces

A comparative assessment of deep learning models for day-ahead load forecasting: Investigating key accuracy drivers

Short-term load forecasting (STLF) is vital for the effective and economic operation of power grids and energy markets. However, the non-linearity and non-stationarity of electricity demand as well as its dependency on various external factors renders STLF a challenging task. To that end, several deep learning models have been proposed in the literature for STLF, reporting promising results. In order to evaluate the accuracy of said models in day-ahead forecasting settings, in this paper we focus on the national net aggregated STLF of Portugal and conduct a comparative study considering a set of indicative, well-established deep autoregressive models, namely multi-layer perceptrons (MLP), long short-term memory networks (LSTM), neural basis expansion coefficient analysis (N-BEATS), temporal convolutional networks (TCN), and temporal fusion transformers (TFT). Moreover, we identify factors that significantly affect the demand and investigate their impact on the accuracy of each model. Our results suggest that N-BEATS consistently outperforms the rest of the examined models. MLP follows, providing further evidence towards the use of feed-forward networks over relatively more sophisticated architectures. Finally, certain calendar and weather features like the hour of the day and the temperature are identified as key accuracy drivers, providing insights regarding the forecasting approach that should be used per case.Comment: Keywords: Short-Term Load Forecasting, Deep Learning, Ensemble, N-BEATS, Temporal Convolution, Forecasting Accurac

Enhancing statistical wind speed forecasting models : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Engineering at Massey University, Manawatū Campus, New Zealand

In recent years, wind speed forecasting models have seen significant development and growth. In particular, hybrid models have been emerging since the last decade. Hybrid models combine two or more techniques from several categories, with each model utilizing its distinct strengths. Mainly, data-driven models that include statistical and Artificial Intelligence/Machine Learning (AI/ML) models are deployed in hybrid models for shorter forecasting time horizons (< 6hrs). Literature studies show that machine learning models have gained enormous potential owing to their accuracy and robustness. On the other hand, only a handful of studies are available on the performance enhancement of statistical models, despite the fact that hybrid models are incomplete without statistical models. To address the knowledge gap, this thesis identified the shortcomings of traditional statistical models while enhancing prediction accuracy. Three statistical models are considered for analyses: Grey Model [GM(1,1)], Markov Chain, and Holt’s Double Exponential Smoothing models. Initially, the problems that limit the forecasting models' applicability are highlighted. Such issues include negative wind speed predictions, failure of predetermined accuracy levels, non-optimal estimates, and additional computational cost with limited performance. To address these concerns, improved forecasting models are proposed considering wind speed data of Palmerston North, New Zealand. Several methodologies have been developed to improve the model performance and fulfill the necessary and sufficient conditions. These approaches include adjusting dynamic moving window, self-adaptive state categorization algorithm, a similar approach to the leave-one-out method, and mixed initialization method. Keeping in view the application of the hybrid methods, novel MODWT-ARIMA-Markov and AGO-HDES models are further proposed as secondary objectives. Also, a comprehensive analysis is presented by comparing sixteen models from three categories, each for four case studies, three rolling windows, and three forecasting horizons. Overall, the improved models showed higher accuracy than their counter traditional models. Finally, the future directions are highlighted that need subsequent research to improve forecasting performance further

Machine Learning for Short-Term Water Demand Predictions

Urban water supply is coming under increased pressure due to urbanisation, water scarcity and climate change. Efficient urban water management can help alleviate this pressure by improving service quality and reducing water loss. Accurate demand and consumption forecasting enables expansion planning, financing, and operation of water distribution systems. Current research often focuses on model-centric approaches where the model is improved to drive forecast accuracy; however, more efficient data usage could be realised as an alternative to model-centric approaches, without incurring additional computation costs. This work investigates the potential of data-centric forecasting approaches, focusing on ways to improve the efficiency of data and computation resource usage for short-term water demand forecasting. To initiate the investigation, several intrinsically different forecasting models are analysed. A total of four different forecasting models, i.e., Prophet, Autoregressive Integrated Moving Average, Neural Network (NN) and Random Forest (RF) are applied to four demand datasets, i.e., one Chinese hourly demand dataset and three UK 15-minute demand datasets. Various aspects of data and model requirements for optimal performance are investigated. Results obtained from the case studies show that prolonging training data may not be necessary, and that accurate sub-daily water demand forecasting is possible with 10 days of past data for model training. In terms of accuracy, neural network and random forest tend to be better suited towards short-term water demand forecasting over statistical models. The second part of the work aims to unbox the four black-box machine learning methods – NN, Long Short-Term Memory (LSTM), RF, Extreme Gradient Boosting (XGB) and explain their inner workings using SHapley Additive exPlanations and Local Interpretable Model-Agnostic Explanations, Prophet and ARIMA are excluded due to inferior forecasting accuracy. Results have found that feature requirement depends on data resolution, the forecasting model used and the forecast time of day. Network-based models (NN and LSTM) are more temporally dependent and feature intensive, indicating that they require more feature inputs to produce equal accuracy compared to tree-based models (RF and XGB). High-resolution forecasts can maintain a high level of accuracy with only one feature at the previous point. The final part of the work analyses the possibility of incorporating Transfer Learning (TL) into the context of water demand forecasting. To evaluate the potential of TL, 18 UK DMAs water demand datasets are used. Experiments are designed to predict water demands in one DMA that has limited or unavailable data, with an aim to anaysing the forecasting ability of models built with alternative DMA data. Results have found that four and eight external DMA datasets are respectively suitable for 15-minute and hourly demand and that limited accuracy gains are achieved from samples size larger than 20,000. Finally, TL-incorporated methods can improve machine learning forecasting accuracy when there is limited data availability. The results obtained in this study prove the usefulness of data-centric approaches’ ability to improve forecasting accuracy. The data-centric approaches explored in this thesis can be used to guide the development of machine learning-based short-term demand forecasting models for researchers, operators, and utilities. Efficient use of forecasting models and demand data holds further potential in improving forecast accuracy, reducing computation cost, and bettering user confidence in the application of machine learning models.EPSR

K-Means and Alternative Clustering Methods in Modern Power Systems

As power systems evolve by integrating renewable energy sources, distributed generation, and electric vehicles, the complexity of managing these systems increases. With the increase in data accessibility and advancements in computational capabilities, clustering algorithms, including K-means, are becoming essential tools for researchers in analyzing, optimizing, and modernizing power systems. This paper presents a comprehensive review of over 440 articles published through 2022, emphasizing the application of K-means clustering, a widely recognized and frequently used algorithm, along with its alternative clustering methods within modern power systems. The main contributions of this study include a bibliometric analysis to understand the historical development and wide-ranging applications of K-means clustering in power systems. This research also thoroughly examines K-means, its various variants, potential limitations, and advantages. Furthermore, the study explores alternative clustering algorithms that can complete or substitute K-means. Some prominent examples include K-medoids, Time-series K-means, BIRCH, Bayesian clustering, HDBSCAN, CLIQUE, SPECTRAL, SOMs, TICC, and swarm-based methods, broadening the understanding and applications of clustering methodologies in modern power systems. The paper highlights the wide-ranging applications of these techniques, from load forecasting and fault detection to power quality analysis and system security assessment. Throughout the examination, it has been observed that the number of publications employing clustering algorithms within modern power systems is following an exponential upward trend. This emphasizes the necessity for professionals to understand various clustering methods, including their benefits and potential challenges, to incorporate the most suitable ones into their studies