Feature Selection and Weighting by Nearest Neighbor Ensembles

In the field of statistical discrimination nearest neighbor methods are a well known, quite simple but successful nonparametric classification tool. In higher dimensions, however, predictive power normally deteriorates. In general, if some covariates are assumed to be noise variables, variable selection is a promising approach. The paper’s main focus is on the development and evaluation of a nearest neighbor ensemble with implicit variable selection. In contrast to other nearest neighbor approaches we are not primarily interested in classification, but in estimating the (posterior) class probabilities. In simulation studies and for real world data the proposed nearest neighbor ensemble is compared to an extended forward/backward variable selection procedure for nearest neighbor classifiers, and some alternative well established classification tools (that offer probability estimates as well). Despite its simple structure, the proposed method’s performance is quite good - especially if relevant covariates can be separated from noise variables. Another advantage of the presented ensemble is the easy identification of interactions that are usually hard to detect. So not simply variable selection but rather some kind of feature selection is performed. The paper is a preprint of an article published in Chemometrics and Intelligent Laboratory Systems. Please use the journal version for citation

Multiple proportion case-basing driven CBRE and its application in the evaluation of possible failure of firms

Case-based reasoning (CBR) is a unique tool for the evaluation of possible failure of firms (EOPFOF) for its eases of interpretation and implementation. Ensemble computing, a variation of group decision in society, provides a potential means of improving predictive performance of CBR-based EOPFOF. This research aims to integrate bagging and proportion case-basing with CBR to generate a method of proportion bagging CBR for EOPFOF. Diverse multiple case bases are first produced by multiple case-basing, in which a volume parameter is introduced to control the size of each case base. Then, the classic case retrieval algorithm is implemented to generate diverse member CBR predictors. Majority voting, the most frequently used mechanism in ensemble computing, is finally used to aggregate outputs of member CBR predictors in order to produce final prediction of the CBR ensemble. In an empirical experiment, we statistically validated the results of the CBR ensemble from multiple case bases by comparing them with those of multivariate discriminant analysis, logistic regression, classic CBR, the best member CBR predictor and bagging CBR ensemble. The results from Chinese EOPFOF prior to 3 years indicate that the new CBR ensemble, which significantly improved CBRs predictive ability, outperformed all the comparative methods

Mechanical and thermal behavior of multiscale bi-nano-composites using experiments and machine learning predictions

The mechanical and thermal properties of natural short latania fiber (SLF)-reinforced poly(propylene)/ethylene-propylene-diene-monomer (SLF/PP/EPDM) bio-composites reinforced with nano-clays (NCs), pistachio shell powders (PSPs), and/or date seed particles (DSPs) were studied using experiments and machine learning (ML) predictions. This dissertation embraces three related investigations: (1) an assessment of maleated polypropylene (MAPP) coupling agent on mechanical and thermal behavior of SLF/PP/EPDM composites, (2) heat deflection temperature (HDT) of bio-nano-composites using experiments and ML predictions, and (3) fracture toughness ML predictions of short fiber, nano- and micro-particle reinforced composites. The first project (Chapter 2) investigates the influence of MAPP on tensile, bending, Charpy impact and HDT of SLF/PP/EPDM composites containing various SLF contents. The second project (Chapter 3) introduces two new bio-powderditives (DSP and PSP) and characterizes the HDT of PP/EPDM composites using experiments and K-Nearest Neighbor Regressor (KNNR) ML predictions. The composites contain various contents of SLF (0, 5, 10, 20, and 30wt%), NCs (0, 1, 3, 5wt%), micro-sized PSPs (0, 1, 3, 5wt%) and micro-sized DSPs (0, 1, 3, 5wt%). The third project (Chapter 4) characterizes the fracture toughness of the same composite series used in the second project, by applying Charpy impact tests, finite element analysis, and a ML approach using the Decision Tree Regressor (DTR) and Adaptive Boosting Regressor (ABR). 2wt% MAPP addition enhanced the composite tensile/flexural moduli and strength up to 9% compared with the composites with zero MAPP. In addition, energy impact absorption was profoundly increased (up to78%) and HDT (up to 4 Co) was improved upon MAPP addition to the composites. SLF, NC, DSP and PSP could separately and conjointly increase HDT and fracture toughness values. The KNNR ML approach could accurately predict the composite’s HDT values and, Decision Tree Regressor (DTR) and Adaptive Boosting Regressor ML algorithms worked well with fracture toughness predictions. Pictures taken through a transmission electron microscope, scanning electron microscope and X-Ray proved the NC dispersion and exfoliation as one of the factors in HDT and fracture toughness improvements

Data-Driven Modeling and Forecasting of Chaotic Dynamics on Inertial Manifolds Constructed as Spectral Submanifolds

We present a data-driven and interpretable approach for reducing the dimensionality of chaotic systems using spectral submanifolds (SSMs). Emanating from fixed points or periodic orbits, these SSMs are low-dimensional inertial manifolds containing the chaotic attractor of the underlying high-dimensional system. The reduced dynamics on the SSMs turn out to predict chaotic dynamics accurately over a few Lyapunov times and also reproduce long-term statistical features, such as the largest Lyapunov exponents and probability distributions, of the chaotic attractor. We illustrate this methodology on numerical data sets including a delay-embedded Lorenz attractor, a nine-dimensional Lorenz model, and a Duffing oscillator chain. We also demonstrate the predictive power of our approach by constructing an SSM-reduced model from unforced trajectories of a buckling beam, and then predicting its periodically forced chaotic response without using data from the forced beam.Comment: Submitted to Chao

Математичне моделювання і прогнозування нелінійних нестаціонарних процесів в економіці та фінансах

Магістерська дисертація: 103 с., 20 рис., 28 табл., 57 джерел, 1 додаток. Актуальність теми. Мінімізація фінансових ризиків та необхідність прийняття обґрунтованих управлінських рішень вимагають вдосконалення існуючих та пошуку нових підходів для реалізації високоякісних прогнозів. Зв’язок роботи з науковими програмами, планами, темами. Дисертаційна робота виконувалась згідно з планом науково-дослідних робіт кафедри математичних методів системного аналізу Національного технічного університету України «Київський політехнічний інститут імені Ігоря Сікорського». Мета дослідження. Підвищення адекватності математичних моделей нелінійних нестаціонарних процесів в економіці та фінансах і якості оцінок їх прогнозів за рахунок створення нових математичних моделей. Об’єкт дослідження. Нелінійні нестаціонарні процеси в економіці та фінансах, представлені статистичними даними стосовно їх розвитку. Предмет дослідження. Математичні моделі аналізу даних з метою моделювання і прогнозування розвитку нелінійних нестаціонарних процесів; множини критеріїв для аналізу адекватності моделей та оцінювання якості прогнозів. Методи дослідження. Використання регресійного підходу та методу групового урахування аргументів, метод найменших квадратів для оцінки параметрів моделі. Наукова новизна отриманих результатів. Розроблено програмний продукт для моделювання та прогнозування нелінійний нестаціонарних процесів. Апробація результатів дисертації. За матеріалами дисертаційної роботи опубліковано 1 тези конференції та подано до друку 1 статтю.The theme: Mathematical modeling and forecasting of nonlinear nonstationary processes in economics and finance. Master’s thesis: 103 p., 20 fig., 28 tab., 57 sources, 1 appendix. Topic Relevance. Minimizing financial risks and the need to make sound management decisions require improving existing and finding new approaches to implementing high-quality forecasts. Thesis connection to scientific programs, plans, and topics. The thesis was prepared according to the scientific research plan of the Department of Mathematical Methods of System Analysis of the National Technical University of Ukraine “Igor Sikorsky Kyiv Polytechnic Institute. Research goal. Improving the adequacy of mathematical models of nonlinear nonstationary processes in economics and finance and the quality of estimates of their forecasts by creating new mathematical models. Object of research. Nonlinear non-stationary processes in economics and finance, represented by statistics on their development. Subject of research. Mathematical models of data analysis for the purpose of modeling and forecasting the development of nonlinear nonstationary processes; sets of criteria for analyzing the adequacy of models and assessing the quality of forecasts. Methods of research. Using a regression approach and the method of group consideration of arguments, the method of least squares to estimate the parameters of the model. Scientific contribution. A software product for modeling and forecasting of nonlinear nonstationary processes has been developed

Previsão de demanda de energia elétrica de curto prazo utilizando abordagens de comitês de Wavenets

Orientador : Prof. Dr. Leandro dos Santos CoelhoDissertação (mestrado) - Universidade Federal do Paraná, Setor de Tecnologia, Programa de Pós-Graduação em Engenharia Elétrica. Defesa: Curitiba, 12/04/2017Inclui referências : f. 87-98Área de concentração: Sistemas eletrônicosResumo: A energia elétrica faz parte de um mercado que envolve agentes de geração, transmissão, distribuição e consumo que desejam maximizar seus lucros e minimizar suas despesas. Para isso precisam de um planejamento que tenha como base uma previsão de demanda precisa, já que um cenário pessimista pode levar ao despacho de mais geradores do que o necessário, reserva excessiva de matéria prima e aumento do custo de operação, e por outro lado um cenário otimista pode colocar o sistema elétrico em risco ou exigir a compra de energia no mercado livre a um preço alto. Por isso, a previsão de demanda tem sido empregada em áreas como o agendamento ótimo de geradores, planejamento da manutenção, planejamento da reserva hídrica, compreensão do padrão de consumo, planejamento da expansão e previsão de preços e ajuste de tarifas. Contudo, uma série de demanda é uma série temporal que possui não linearidades e componentes periódicos aleatórios como o clima, perfil dos usuários, eventos públicos, economia, medições erradas, e, consequentemente, um modelo de previsão linear pode não ser apropriado. Este trabalho utiliza diferentes abordagens para formar comitês de wavenets para a previsão de séries temporais de demanda de energia elétrica de curto prazo, os desempenhos são comparados com uma rede neural artificial perceptron multicamadas com função de ativação sigmoide na camada oculta, uma wavenet simples, com a média da última semana e com o modelo inocente. As séries de demanda adotadas, isto é, duas séries de demanda anuais reais com medições horárias, passam por um estágio de pré-processamento para remoção da tendência e normalização, e também para transformação dos valores da série em conjuntos de entrada e saída para o treinamento supervisionado. Emprega-se a estratégia de previsão um passo à frente e a avaliação das previsões é realizada pelo coeficiente de correlação múltipla ???? e também pela análise de correlação entre os resíduos. Para criação dos comitês utiliza-se a reamostragem com reposição, a validação cruzada e a dizimação de entradas, seleção construtiva, combinação pela média simples, moda, mediana e generalização empilhada. Os resultados dos testes de não linearidade demonstram que as duas séries consideradas são não lineares, e também constata-se a diminuição da assimetria dos dados após sua transformação. Do processo de seleção de variáveis obtém-se os atrasos máximos para cada série, valores passados que são utilizados como entradas, e percebe-se que são diferentes para cada série. O atraso máximo a ser utilizado como entrada tem influência na quantidade de amostras do conjunto de dados de entrada e saída. Uma característica dos resultados que se reflete em ambas as séries é o aumento do erro à medida que o horizonte de previsão aumenta. Os comitês de wavenets superam os demais modelos comparados, e, além do desempenho ser diferente para cada problema, o melhor método de aprendizado de comitê a ser utilizado também varia, bem como o horizonte de previsão máximo no qual os valores previstos se ajustam aos valores reais das séries. A qualidade das previsões é avaliada com testes de correlação dos resíduos. Palavras-chave: Wavenet. Previsão de demanda de energia elétrica. Comitês. Redes neurais artificiais.Abstract: Electricity is part of a market which involves generation, transmission, distribution and consumption agents that aim their profit maximization and expenses minimization. To achieve that, they need a planning based on an accurate load forecast, since a pessimistic scenario may lead to more generators dispatch than needed, excessive reservoir and high operating costs, and, on the other hand, an optimistic scenario may place the electrical system at risk or requiring demand electricity purchasing on the free market for a very high price. Hence, load forecasting has been employed in areas such as optimal dispatch, maintenance planning, hydric reservoir planning, consumption pattern understanding, expansion planning, price forecasting and tax adjustments. However, a load series is a time series with nonlinearities and random periodic components as the weather, users profile, public events, economy and bad measures, therefore a pure linear model may not be appropriated. This work uses different approaches to create wavenet ensembles for short term load forecasting, the performances are compared with a multilayer perceptron with sigmoid activation function in the hidden layer, with a single wavenet, with the last week mean and also with the naive model. The load series adopted, that is, two annual hourly load series with actual measurements, are passed through a data pre-processing stage for trend removal and normalization, and also for conversion from the time series to a inputs and output set for supervised training. It is applied the one step ahead forecast strategy and the forecasting evaluation is accomplished by the multiple correlation coefficient, ????, and also by the residuals correlation analysis. For the ensemble creation are used the bootstrapping, cross-validation like, inputs decimation, constructive selection, simple average, median, mode and stacked generalization methods. The nonlinearity tests results demonstrate that both time series are nonlinear, and the asymmetry reduction after data transformation is verified. From the features selection process the maximum lags for each series are identified, lagged values to be used as inputs and it is noticed that they are different for each series. The maximum lag also influences the amount of samples in the dataset of inputs and outputs. A common characteristic of both series is that the error increase along with the prediction horizon. Results point out that the wavenets ensembles overcome the other compared models after tests with two actual annual hourly load series. Moreover, beyond the performance to be different for each problem, the best ensemble learning method also varies, as well as the maximum forecasting horizon for which the forecasted values fit the series actual values. The quality of the forecasts is analyzed through residuals correlation tests. Key-words: Wavenet. Load forecasting. Ensembles. Artificial neural network

Forecasting for Nonlinear and Nonstationary Systems Using Intrinsic Functional Decomposition Models

The purpose of this study is to develop nonlinear and nonstationary time series forecasting methods to address modeling and prediction of real-world, complex systems. Particular emphasis has been placed on nonlinear and nonstationary time series forecasting in systems and processes that are of interest to IE researchers. Two new advanced prediction methods are developed using nonlinear decomposition techniques and a battery of advanced statistical methods. The research methodologies include empirical mode decomposition (EMD)-based prediction, structural relationship identification (SRI) methodology, and intrinsic time-scale decomposition (ITD)-based prediction. The advantages of using these prediction methods are local characteristic time scales and the use of an adaptive basis that does not require a parametric functional form (during the decomposition process). The utilization of SRI methodology in ITD-based prediction also provides a relationship identification advantage that can be used to capture the interrelationships of variables in the system for prediction application. The empirical results of using these new prediction methods have shown a significant improvement in the accuracy for customer willingness-to-pay and automobile demand prediction applications.Industrial Engineering & Managemen

Métodos de predicción de la generación agragada de energía eólica

El fomento y desarrollo de las energías renovables, y como consecuencia la alta penetración eólica en los sistemas eléctricos, ha hecho necesario el uso de herramientas de predicción, para conocer con antelación suficiente la cantidad de energía de origen eólico que será inyectada en la red y poder coordinar el resto de fuentes de generación. En ocasiones existe un especial interés en la predicción de generación agregada de varios parques en una región, ya sea por parte del operador del sistema eléctrico, o por los agentes que gestionan la energía que generan varios parques. Por otro lado, suele haber un suavizado espacial que reduce el error de las predicciones agregadas frente al error de predicción de los parques individuales. Estos motivos hacen de la predicción agregada un modo muy atractivo de predicción eólica. En esta tesis se propone un método de predicción de la generación agregada de energía eólica en una región, mediante la búsqueda de similitudes entre el vector de viento previsto en algunas coordenadas de la región y otros vectores históricos de predicción de viento. El modelo propuesto se basa en modelos locales de suavizado de Media Ponderada y de Regresión Local Ponderada. Estos modelos ajustan de forma no paramétrica un modelo de predicción para cada punto de entrada al problema, que en nuestro caso será un vector de viento previsto. Para hacer el suavizado de los datos, se proponen y comparan distintos procedimientos para tener en cuenta la medida de distancias entre vectores, el método de selección de datos cercanos, y las funciones de ponderación por distancia y antigüedad de los datos seleccionados. Mediante un procedimiento de estimación adaptativa de parámetros, se conseguirá que el modelo de predicción se adecue a la evolución temporal de la compleja relación no lineal existente entre el viento previsto en la región y la potencia eólica total generada. Las predicciones de potencia obtenidas con los modelos de Media Ponderada y Regresión Local Ponderada, se combinan para aprovechar las ventajas que ofrecen ambas aproximaciones en función de la complejidad del problema de predicción en cada momento. El modelo propuesto se validará comparando sus predicciones con la agregación de predicciones que se obtienen con Sipreólico, una herramienta que calcula predicciones para cada uno de los parques en la región y después las suma. Por último, para dar mayor valor a las predicciones agregadas, se propone un modelo de predicción probabilista condicionada a los vectores de viento previsto mediante estimación de densidades con Kernel, para el cual se utilizan las medidas de distancia y funciones de ponderación propuestas para el modelo de predicción de potencia.---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------The promotion and development of renewable energy resources, and as a result, the high wind power penetration in the power systems, makes necessary the use of forecasting tools, in order to know in advance the amount of wind energy that will be injected into the network and to coordinate all the other generation sources. Sometimes, there is a particular interest in predicting the aggregated wind power in a region, either by the system operator or other practitioners who manage the energy generated by several wind farms. On the other hand, there is usually a spatial smoothing effect that reduces the prediction error when making aggregated wind power predictions, instead of predicting individual wind farms. These reasons make the aggregated wind power prediction models very appealing. This thesis proposes a method to make aggregated wind power predictions in a region, by seeking similarities between the next wind speed forecasting vector and a set of historical vectors of wind speed in some coordinates of the region. The proposed model is based on Weighted Average and Locally Weighted Regression smoothing models. These models fit a non-parametric prediction model for each query point, which in our case is a vector containing the wind speed forecasts for a set of coordinates in the region. In order to smooth the data, different smoothing procedures are proposed and compared. These procedures must take into account the measure of distances between wind speed vectors, the nearby data selection method, and the distance and age weighting functions for the selected data. An adaptive estimation procedure of the parameters will let the model to fit the timevarying evolution of the complex nonlinear relationship between the expected wind speed in the coordinates and the total wind power generated in the region. The wind power predictions obtained with the alternative models, based on Weighted Average and Locally Weighted Regression, are combined in order to exploit the advantages of both approaches depending on the complexity of the prediction problem in every moment. The proposed model is validated by comparing its predictions with the aggregation of predictions obtained with Sipreólico, a wind power prediction tool that makes predictions for each of the wind farms in the region and adds them. Finally, more value is added to the aggregated model by proposing a method to make probabilistic forecasts conditioned to the expected wind speed vectors using Kernel density estimation. This approach uses the measures of distance and the weighting functions proposed for the wind power prediction model

Ensembles of Nearest Neighbor Forecasts

Abstract. Nearest neighbor forecasting models are attractive with their simplicity and the ability to predict complex nonlinear behavior. They rely on the assumption that observations similar to the target one are also likely to have similar outcomes. A common practice in nearest neighbor model selection is to compute the globally optimal number of neighbors on a validation set, which is later applied for all incoming queries. For certain queries, however, this number may be suboptimal and forecasts that deviate a lot from the true realization could be produced. To address the problem we propose an alternative approach of training ensembles of nearest neighbor predictors that determine the best number of neighbors for individual queries. We demonstrate that the forecasts of the ensembles improve significantly on the globally optimal single predictors.