Meta-learning for time series forecasting and forecast combination

In research of time series forecasting, a lot of uncertainty is still related to the task of selecting an appropriate forecasting method for a problem. It is not only the individual algorithms that are available in great quantities; combination approaches have been equally popular in the last decades. Alone the question of whether to choose the most promising individual method or a combination is not straightforward to answer. Usually, expert knowledge is needed to make an informed decision, however, in many cases this is not feasible due to lack of resources like time, money and manpower. This work identifies an extensive feature pool describing both the time series and the ensemble of individual forecasting methods. The applicability of different meta-learning approaches are investigated, first to gain knowledge on which model works best in which situation, later to improve forecasting performance. Results show the superiority of a ranking-based combination of methods over simple model selection approaches

Encountered Problems of Time Series with Neural Networks: Models and Architectures

The growing interest in the development of forecasting applications with neural networks is denoted by the publication of more than 10,000 research articles present in the literature. However, the high number of factors included in the configuration of the network, the training process, validation and forecasting, and the sample of data, which must be determined in order to achieve an adequate network model for forecasting, converts neural networks in an unstable technique, given that any change in training or in some parameter produces great changes in the prediction. In this chapter, an analysis of the problematic around the factors that affect the construction of the neural network models is made and that often present inconsistent results, and the fields that require additional research are highlighted

An Evaluation of Methods for Combining Univariate Time Series Forecasts

This thesis presents and evaluates nineteen methods for combining up to eleven automated univariate forecasts. The evaluation is made by applying the methods on a dataset containing more than 1000 monthly time series. The accuracy of one period ahead forecasts is analyzed. Almost 3.2 million forecasts are evaluated in the study. Methods that are using past forecasts to optimally produce a combined forecast are included, along with methods that do not require this information. A pre-screening procedure to get rid of the poorest performing forecasting methods before the remaining ones are combined is evaluated. The results confirm that it is possible to achieve a superior forecast accuracy by combining forecasts. The best methods that utilize past forecasts tend to outperform the best methods that are not considering this data. Including a pre-screening procedure to remove inferior forecasts before combining forecasts from the top five ranked methods seems to increase the forecast accuracy. The pre-screening procedure consists of ranking the automated univariate forecasting methods using an independent, but relevant, dataset. The four best performing methods utilize the pre-screening procedure together with past forecasts to optimally combine forecasts. The best method computes the historical mean squared error of each individual method and weights them accordingly. Demand for automated procedures is growing as the size of datasets increases within organizations. Forecasting from a large set of time series is an activity that can take advantage of automated procedures. However, choosing which forecasting method to use is often problematic. One way of solving this is by combining multiple forecasts into a single forecast

Combinations of time series forecasts: when and why are they beneficial?.

Time series forecasting has a long track record in many application areas. In forecasting research, it has been illustrated that finding an individual algorithm that works best for all possible scenarios is hopeless. Therefore, instead of striving to design a single superior algorithm, current research efforts have shifted towards gaining a deeper understanding of the reasons a forecasting method may perform well in some conditions whilst it may fail in others. This thesis provides a number of contributions to this matter. Traditional empirical evaluations are discussed from a novel point of view, questioning the benefit of using sophisticated forecasting methods without domain knowledge. An own empirical study focusing on relevant off-the shelf forecasting and forecast combination methods underlines the competitiveness of relatively simple methods in practical applications. Furthermore, meta-features of time series are extracted to automatically find and exploit a link between application specific data characteristics and forecasting performance using meta-learning. Finally, the approach of extending the set of input forecasts by diversifying functional approaches, parameter sets and data aggregation level used for learning is discussed, relating characteristics of the resulting forecasts to different error decompositions for both individual methods and combinations. Advanced combination structures are investigated in order to take advantage of the knowledge on the forecast generation processes. Forecasting is a crucial factor in airline revenue management; forecasting of the anticipated booking, cancellation and no-show numbers has a direct impact on general planning of routes and schedules, capacity control for fareclasses and overbooking limits. In a collaboration with Lufthansa Systems in Berlin, experiments in the thesis are conducted on an airline data set with the objective of improving the current net booking forecast by modifying one of its components, the cancellation forecast. To also compare results achieved of the methods investigated here with the current state-of-the-art in forecasting research, some experiments also use data sets of two recent forecasting competitions, thus being able to provide a link between academic research and industrial practice

Selecting and Ranking Time Series Models Using the NOEMON Approach

Abstract. In this work, we proposed to use the NOEMON approach to rank and select time series models. Given a time series, the NOEMON approach provides a ranking of the candidate models to forecast that series, by combining the outputs of different learners. The best ranked models are then returned as the selected ones. In order to evaluate the proposed solution, we implemented a prototype that used MLP neural networks as the learners. Our experiments using this prototype revealed encouraging results.

Financial predictions using intelligent systems

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.This thesis presents a collection of practical techniques for analysing various market properties in order to design advanced self-evolving trading systems based on neural networks combined with a genetic algorithm optimisation approach. Nonlinear multivariate statistical models have gained increasing importance in financial time series analysis, as it is very hard to fmd statistically significant market inefficiencies using standard linear modes. Nonlinear models capture more of the underlying dynamics of these high dimensional noisy systems than traditional models, whilst at the same time making fewer restrictive assumptions about them. These adaptive trading systems can extract information about associated time varying processes that may not be readily captured by traditional models. In order to characterise the fmancial time series in terms of its dynamic nature, this research employs various methods such as fractal analysis, chaos theory and dynamical recurrence analysis. These techniques are used for evaluating whether markets are stochastic and deterministic or nonlinear and chaotic, and to discover regularities that are completely hidden in these time series and not detectable using conventional analysis. Particular emphasis is placed on examining the feasibility of prediction in fmancial time series and the analysis of extreme market events. The market's fractal structure and log-periodic oscillations, typical of periods before extreme events occur, are revealed through recurrence plots. Recurrence qualification analysis indicated a strong presence of structure, recurrence and determinism in the fmancial time series studied. Crucial fmancial time series transition periods were also detected. This research performs several tests on a large number of US and European stocks using methodologies inspired by both fundamental analysis and technical trading rules. Results from the tests show that profitable trading models utilising advanced nonlinear trading systems can be created after accounting for realistic transaction costs. The return achieved by applying the trading model to a portfolio of real price series differs significantly from that achieved by applying it to a randomly generated price series. In some cases, these models are compared against simpler alternative approaches to ensure that there is an added value in the use of these more complex models. The superior performance of multivariate nonlinear models is also demonstrated. The long-short trading strategies performed well in both bull and bear markets, as well as in a sideways market, showing a great degree of flexibility and adjustability to changing market conditions. Empirical evidence shows that information is not instantly incorporated into market pnces and supports the claim that the fmancial time series studied, for the periods analysed, are not entirely random. This research clearly shows that equity markets are partially inefficient and do not behave along lines dictated by the efficient market hypothesis

A Review of Meta-level Learning in the Context of Multi-component, Multi-level Evolving Prediction Systems.

The exponential growth of volume, variety and velocity of data is raising the need for investigations of automated or semi-automated ways to extract useful patterns from the data. It requires deep expert knowledge and extensive computational resources to find the most appropriate mapping of learning methods for a given problem. It becomes a challenge in the presence of numerous configurations of learning algorithms on massive amounts of data. So there is a need for an intelligent recommendation engine that can advise what is the best learning algorithm for a dataset. The techniques that are commonly used by experts are based on a trial and error approach evaluating and comparing a number of possible solutions against each other, using their prior experience on a specific domain, etc. The trial and error approach combined with the expert’s prior knowledge, though computationally and time expensive, have been often shown to work for stationary problems where the processing is usually performed off-line. However, this approach would not normally be feasible to apply on non-stationary problems where streams of data are continuously arriving. Furthermore, in a non-stationary environment the manual analysis of data and testing of various methods every time when there is a change in the underlying data distribution would be very difficult or simply infeasible. In that scenario and within an on-line predictive system, there are several tasks where Meta-learning can be used to effectively facilitate best recommendations including: 1) pre processing steps, 2) learning algorithms or their combination, 3) adaptivity mechanisms and their parameters, 4) recurring concept extraction, and 5) concept drift detection. However, while conceptually very attractive and promising, the Meta-learning leads to several challenges with the appropriate representation of the problem at a meta-level being one of the key ones. The goal of this review and our research is, therefore, to investigate Meta learning in general and the associated challenges in the context of automating the building, deployment and adaptation of multi-level and multi-component predictive system that evolve over time

Accommodating maintenance in prognostics

Error on title page - year of award is 2021Steam turbines are an important asset of nuclear power plants, and are required to operate reliably and efficiently. Unplanned outages have a significant impact on the ability of the plant to generate electricity. Therefore, condition-based maintenance (CBM) can be used for predictive and proactive maintenance to avoid unplanned outages while reducing operating costs and increasing the reliability and availability of the plant. In CBM, the information gathered can be interpreted for prognostics (the prediction of failure time or remaining useful life (RUL)). The aim of this project was to address two areas of challenges in prognostics, the selection of predictive technique and accommodation of post-maintenance effects, to improve the efficacy of prognostics. The selection of an appropriate predictive algorithm is a key activity for an effective development of prognostics. In this research, a formal approach for the evaluation and selection of predictive techniques is developed to facilitate a methodic selection process of predictive techniques by engineering experts. This approach is then implemented for a case study provided by the engineering experts. Therefore, as a result of formal evaluation, a probabilistic technique the Bayesian Linear Regression (BLR) and a non-probabilistic technique the Support Vector Regression (SVR) were selected for prognostics implementation. In this project, the knowledge of prognostics implementation is extended by including post maintenance affects into prognostics. Maintenance aims to restore a machine into a state where it is safe and reliable to operate while recovering the health of the machine. However, such activities result in introduction of uncertainties that are associated with predictions due to deviations in degradation model. Thus, affecting accuracy and efficacy of predictions. Therefore, such vulnerabilities must be addressed by incorporating the information from maintenance events for accurate and reliable predictions. This thesis presents two frameworks which are adapted for probabilistic and non-probabilistic prognostic techniques to accommodate maintenance. Two case studies: a real-world case study from a nuclear power plant in the UK and a synthetic case study which was generated based on the characteristics of a real-world case study are used for the implementation and validation of the frameworks. The results of the implementation hold a promise for predicting remaining useful life while accommodating maintenance repairs. Therefore, ensuring increased asset availability with higher reliability, maintenance cost effectiveness and operational safety.Steam turbines are an important asset of nuclear power plants, and are required to operate reliably and efficiently. Unplanned outages have a significant impact on the ability of the plant to generate electricity. Therefore, condition-based maintenance (CBM) can be used for predictive and proactive maintenance to avoid unplanned outages while reducing operating costs and increasing the reliability and availability of the plant. In CBM, the information gathered can be interpreted for prognostics (the prediction of failure time or remaining useful life (RUL)). The aim of this project was to address two areas of challenges in prognostics, the selection of predictive technique and accommodation of post-maintenance effects, to improve the efficacy of prognostics. The selection of an appropriate predictive algorithm is a key activity for an effective development of prognostics. In this research, a formal approach for the evaluation and selection of predictive techniques is developed to facilitate a methodic selection process of predictive techniques by engineering experts. This approach is then implemented for a case study provided by the engineering experts. Therefore, as a result of formal evaluation, a probabilistic technique the Bayesian Linear Regression (BLR) and a non-probabilistic technique the Support Vector Regression (SVR) were selected for prognostics implementation. In this project, the knowledge of prognostics implementation is extended by including post maintenance affects into prognostics. Maintenance aims to restore a machine into a state where it is safe and reliable to operate while recovering the health of the machine. However, such activities result in introduction of uncertainties that are associated with predictions due to deviations in degradation model. Thus, affecting accuracy and efficacy of predictions. Therefore, such vulnerabilities must be addressed by incorporating the information from maintenance events for accurate and reliable predictions. This thesis presents two frameworks which are adapted for probabilistic and non-probabilistic prognostic techniques to accommodate maintenance. Two case studies: a real-world case study from a nuclear power plant in the UK and a synthetic case study which was generated based on the characteristics of a real-world case study are used for the implementation and validation of the frameworks. The results of the implementation hold a promise for predicting remaining useful life while accommodating maintenance repairs. Therefore, ensuring increased asset availability with higher reliability, maintenance cost effectiveness and operational safety

Combinations of time series forecasts : when and why are they beneficial?

Time series forecasting has a long track record in many application areas. In forecasting research, it has been illustrated that finding an individual algorithm that works best for all possible scenarios is hopeless. Therefore, instead of striving to design a single superior algorithm, current research efforts have shifted towards gaining a deeper understanding of the reasons a forecasting method may perform well in some conditions whilst it may fail in others. This thesis provides a number of contributions to this matter. Traditional empirical evaluations are discussed from a novel point of view, questioning the benefit of using sophisticated forecasting methods without domain knowledge. An own empirical study focusing on relevant off-the shelf forecasting and forecast combination methods underlines the competitiveness of relatively simple methods in practical applications. Furthermore, meta-features of time series are extracted to automatically find and exploit a link between application specific data characteristics and forecasting performance using meta-learning. Finally, the approach of extending the set of input forecasts by diversifying functional approaches, parameter sets and data aggregation level used for learning is discussed, relating characteristics of the resulting forecasts to different error decompositions for both individual methods and combinations. Advanced combination structures are investigated in order to take advantage of the knowledge on the forecast generation processes. Forecasting is a crucial factor in airline revenue management; forecasting of the anticipated booking, cancellation and no-show numbers has a direct impact on general planning of routes and schedules, capacity control for fareclasses and overbooking limits. In a collaboration with Lufthansa Systems in Berlin, experiments in the thesis are conducted on an airline data set with the objective of improving the current net booking forecast by modifying one of its components, the cancellation forecast. To also compare results achieved of the methods investigated here with the current state-of-the-art in forecasting research, some experiments also use data sets of two recent forecasting competitions, thus being able to provide a link between academic research and industrial practice.EThOS - Electronic Theses Online ServiceGBUnited Kingdo

Meta-level learning for the effective reduction of model search space.

The exponential growth of volume, variety and velocity of the data is raising the need for investigation of intelligent ways to extract useful patterns from the data. It requires deep expert knowledge and extensive computational resources to find the mapping of learning methods that leads to the optimized performance on a given task. Moreover, numerous configurations of these learning algorithms add another level of complexity. Thus, it triggers the need for an intelligent recommendation engine that can advise the best learning algorithm and its configurations for a given task. The techniques that are commonly used by experts are; trial-and-error, use their prior experience on the specific domain, etc. These techniques sometimes work for less complex tasks that require thousands of parameters to learn. However, the state-of-the-art models, e.g. deep learning models, require well-tuned hyper-parameters to learn millions of parameters which demand specialized skills and numerous computationally expensive and time-consuming trials. In that scenario, Meta-level learning can be a potential solution that can recommend the most appropriate options efficiently and effectively regardless of the complexity of data. On the contrary, Meta-learning leads to several challenges; the most critical ones being model selection and hyper-parameter optimization. The goal of this research is to investigate model selection and hyper-parameter optimization approaches of automatic machine learning in general and the challenges associated with them. In machine learning pipeline there are several phases where Meta-learning can be used to effectively facilitate the best recommendations including 1) pre-processing steps, 2) learning algorithm or their combination, 3) adaptivity mechanism parameters, 4) recurring concept extraction, and 5) concept drift detection. The scope of this research is limited to feature engineering for problem representation, and learning strategy for algorithm and its hyper-parameters recommendation at Meta-level. There are three studies conducted around the two different approaches of automatic machine learning which are model selection using Meta-learning and hyper-parameter optimization. The first study evaluates the situation in which the use of additional data from a different domain can improve the performance of a meta-learning system for time-series forecasting, with focus on cross- domain Meta-knowledge transfer. Although the experiments revealed limited room for improvement over the overall best base-learner, the meta-learning approach turned out to be a safe choice, minimizing the risk of selecting the least appropriate base-learner. There are only 2% of cases recommended by meta- learning that are the worst performing base-learning methods. The second study proposes another efficient and accurate domain adaption approach but using a different meta-learning approach. This study empirically confirms the intuition that there exists a relationship between the similarity of the two different tasks and the depth of network needed to fine-tune in order to achieve accuracy com- parable with that of a model trained from scratch. However, the approach is limited to a single hyper-parameter which is fine-tuning of the network depth based on task similarity. The final study of this research has expanded the set of hyper-parameters while implicitly considering task similarity at the intrinsic dynamics of the training process. The study presents a framework to automatically find a good set of hyper-parameters resulting in reasonably good accuracy, by framing the hyper-parameter selection and tuning within the reinforcement learning regime. The effectiveness of a recommended tuple can be tested very quickly rather than waiting for the network to converge. This approach produces accuracy close to the state-of-the-art approach and is found to be comparatively 20% less computationally expensive than previous approaches. The proposed methods in these studies, belonging to different areas of automatic machine learning, have been thoroughly evaluated on a number of benchmark datasets which confirmed the great potential of these methods