Linear and Order Statistics Combiners for Pattern Classification

Several researchers have experimentally shown that substantial improvements can be obtained in difficult pattern recognition problems by combining or integrating the outputs of multiple classifiers. This chapter provides an analytical framework to quantify the improvements in classification results due to combining. The results apply to both linear combiners and order statistics combiners. We first show that to a first order approximation, the error rate obtained over and above the Bayes error rate, is directly proportional to the variance of the actual decision boundaries around the Bayes optimum boundary. Combining classifiers in output space reduces this variance, and hence reduces the "added" error. If N unbiased classifiers are combined by simple averaging, the added error rate can be reduced by a factor of N if the individual errors in approximating the decision boundaries are uncorrelated. Expressions are then derived for linear combiners which are biased or correlated, and the effect of output correlations on ensemble performance is quantified. For order statistics based non-linear combiners, we derive expressions that indicate how much the median, the maximum and in general the ith order statistic can improve classifier performance. The analysis presented here facilitates the understanding of the relationships among error rates, classifier boundary distributions, and combining in output space. Experimental results on several public domain data sets are provided to illustrate the benefits of combining and to support the analytical results.Comment: 31 page

Combining neural network regression estimates with regularized linear weights

When combining a set of learned models to form an improved estimator, the issue of redundancy or multicollinearity in the set of models must be addressed. A progression of existing approaches and their limitations with respect to the redundancy is discussed. A new approach, PCR*, based on principal components regression is proposed to address these limitations. An evaluation of the new approach on a collection of domains reveals that: 1) PCR* was the most robust combination method as the redundancy of the learned models increased, 2) redundancy could be handled without eliminating any of the learned models, and 3) the principal components of the learned models provided a continuum of \regularized" weights from which PCR * could choose.

Previsão de demanda de energia elétrica de curto prazo utilizando abordagens de comitês de Wavenets

Orientador : Prof. Dr. Leandro dos Santos CoelhoDissertação (mestrado) - Universidade Federal do Paraná, Setor de Tecnologia, Programa de Pós-Graduação em Engenharia Elétrica. Defesa: Curitiba, 12/04/2017Inclui referências : f. 87-98Área de concentração: Sistemas eletrônicosResumo: A energia elétrica faz parte de um mercado que envolve agentes de geração, transmissão, distribuição e consumo que desejam maximizar seus lucros e minimizar suas despesas. Para isso precisam de um planejamento que tenha como base uma previsão de demanda precisa, já que um cenário pessimista pode levar ao despacho de mais geradores do que o necessário, reserva excessiva de matéria prima e aumento do custo de operação, e por outro lado um cenário otimista pode colocar o sistema elétrico em risco ou exigir a compra de energia no mercado livre a um preço alto. Por isso, a previsão de demanda tem sido empregada em áreas como o agendamento ótimo de geradores, planejamento da manutenção, planejamento da reserva hídrica, compreensão do padrão de consumo, planejamento da expansão e previsão de preços e ajuste de tarifas. Contudo, uma série de demanda é uma série temporal que possui não linearidades e componentes periódicos aleatórios como o clima, perfil dos usuários, eventos públicos, economia, medições erradas, e, consequentemente, um modelo de previsão linear pode não ser apropriado. Este trabalho utiliza diferentes abordagens para formar comitês de wavenets para a previsão de séries temporais de demanda de energia elétrica de curto prazo, os desempenhos são comparados com uma rede neural artificial perceptron multicamadas com função de ativação sigmoide na camada oculta, uma wavenet simples, com a média da última semana e com o modelo inocente. As séries de demanda adotadas, isto é, duas séries de demanda anuais reais com medições horárias, passam por um estágio de pré-processamento para remoção da tendência e normalização, e também para transformação dos valores da série em conjuntos de entrada e saída para o treinamento supervisionado. Emprega-se a estratégia de previsão um passo à frente e a avaliação das previsões é realizada pelo coeficiente de correlação múltipla ???? e também pela análise de correlação entre os resíduos. Para criação dos comitês utiliza-se a reamostragem com reposição, a validação cruzada e a dizimação de entradas, seleção construtiva, combinação pela média simples, moda, mediana e generalização empilhada. Os resultados dos testes de não linearidade demonstram que as duas séries consideradas são não lineares, e também constata-se a diminuição da assimetria dos dados após sua transformação. Do processo de seleção de variáveis obtém-se os atrasos máximos para cada série, valores passados que são utilizados como entradas, e percebe-se que são diferentes para cada série. O atraso máximo a ser utilizado como entrada tem influência na quantidade de amostras do conjunto de dados de entrada e saída. Uma característica dos resultados que se reflete em ambas as séries é o aumento do erro à medida que o horizonte de previsão aumenta. Os comitês de wavenets superam os demais modelos comparados, e, além do desempenho ser diferente para cada problema, o melhor método de aprendizado de comitê a ser utilizado também varia, bem como o horizonte de previsão máximo no qual os valores previstos se ajustam aos valores reais das séries. A qualidade das previsões é avaliada com testes de correlação dos resíduos. Palavras-chave: Wavenet. Previsão de demanda de energia elétrica. Comitês. Redes neurais artificiais.Abstract: Electricity is part of a market which involves generation, transmission, distribution and consumption agents that aim their profit maximization and expenses minimization. To achieve that, they need a planning based on an accurate load forecast, since a pessimistic scenario may lead to more generators dispatch than needed, excessive reservoir and high operating costs, and, on the other hand, an optimistic scenario may place the electrical system at risk or requiring demand electricity purchasing on the free market for a very high price. Hence, load forecasting has been employed in areas such as optimal dispatch, maintenance planning, hydric reservoir planning, consumption pattern understanding, expansion planning, price forecasting and tax adjustments. However, a load series is a time series with nonlinearities and random periodic components as the weather, users profile, public events, economy and bad measures, therefore a pure linear model may not be appropriated. This work uses different approaches to create wavenet ensembles for short term load forecasting, the performances are compared with a multilayer perceptron with sigmoid activation function in the hidden layer, with a single wavenet, with the last week mean and also with the naive model. The load series adopted, that is, two annual hourly load series with actual measurements, are passed through a data pre-processing stage for trend removal and normalization, and also for conversion from the time series to a inputs and output set for supervised training. It is applied the one step ahead forecast strategy and the forecasting evaluation is accomplished by the multiple correlation coefficient, ????, and also by the residuals correlation analysis. For the ensemble creation are used the bootstrapping, cross-validation like, inputs decimation, constructive selection, simple average, median, mode and stacked generalization methods. The nonlinearity tests results demonstrate that both time series are nonlinear, and the asymmetry reduction after data transformation is verified. From the features selection process the maximum lags for each series are identified, lagged values to be used as inputs and it is noticed that they are different for each series. The maximum lag also influences the amount of samples in the dataset of inputs and outputs. A common characteristic of both series is that the error increase along with the prediction horizon. Results point out that the wavenets ensembles overcome the other compared models after tests with two actual annual hourly load series. Moreover, beyond the performance to be different for each problem, the best ensemble learning method also varies, as well as the maximum forecasting horizon for which the forecasted values fit the series actual values. The quality of the forecasts is analyzed through residuals correlation tests. Key-words: Wavenet. Load forecasting. Ensembles. Artificial neural network

Artificial immune systems based committee machine for classification application

This thesis was submitted for the degree of Doctor of Philosophy and awarded by Brunel University.A new adaptive learning Artificial Immune System (AIS) based committee machine is developed in this thesis. The new proposed approach efficiently tackles the general problem of clustering high-dimensional data. In addition, it helps on deriving useful decision and results related to other application domains such classification and prediction. Artificial Immune System (AIS) is a branch of computational intelligence field inspired by the biological immune system, and has gained increasing interest among researchers in the development of immune-based models and techniques to solve diverse complex computational or engineering problems. This work presents some applications of AIS techniques to health problems, and a thorough survey of existing AIS models and algorithms. The main focus of this research is devoted to building an ensemble model integrating different AIS techniques (i.e. Artificial Immune Networks, Clonal Selection, and Negative Selection) for classification applications to achieve better classification results. A new AIS-based ensemble architecture with adaptive learning features is proposed by integrating different learning and adaptation techniques to overcome individual limitations and to achieve synergetic effects through the combination of these techniques. Various techniques related to the design and enhancements of the new adaptive learning architecture are studied, including a neuro-fuzzy based detector and an optimizer using particle swarm optimization method to achieve enhanced classification performance. An evaluation study was conducted to show the performance of the new proposed adaptive learning ensemble and to compare it to alternative combining techniques. Several experiments are presented using different medical datasets for the classification problem and findings and outcomes are discussed. The new adaptive learning architecture improves the accuracy of the ensemble. Moreover, there is an improvement over the existing aggregation techniques. The outcomes, assumptions and limitations of the proposed methods with its implications for further research in this area draw this research to its conclusion

Desarrollo software de técnicas para el diseño automático de redes neuronales artificiales con bajo coste computacional

Las técnicas de inteligencia computacional, entre las que destacan las Redes Neuronales Artificiales (RNAs), han sido utilizadas con éxito en multitud de contextos y problemas procedentes de la ingeniería de control, la economía, la medicina, etc. Sin embargo, generalmente, las RNAs presentan diversos inconvenientes durante el proceso de optimización de los hiper-parámetros (i.e., pesos y número de neuronas), como son el elevado tiempo computacional requerido y la convergencia a soluciones sub-optimas (mínimos locales) para un determinado problema al usar técnicas de optimización basadas en gradiente. Recientemente, el algoritmo de entrenamiento conocido como Extreme Learning Machine ha permitido solventar estos claros inconvenientes que presentan las técnicas tradicionales de entrenamiento de RNA. En este PFC se desarrollan nuevas técnicas computacionales para el diseño y la combinación automática de RNAs utilizando el novedoso y eficiente algoritmo ELM. Implementando un conjunto de funciones y herramientas software, en el entorno de programación MATLAB, que permitan diseñar automáticamente RNAs con altas prestaciones en términos predictivos y, al mismo tiempo, con bajo coste computacional. Hay que resaltar que este PFC forma parte de un trabajo de investigación de un mayor alcance, donde se están desarrollando nuevos y eficientes sistemas y algoritmos de inteligencia computacional, que está siendo desarrollado fundamentalmente en el Centro Universitario de la Defensa (CUD) de San Javier.Escuela Técnica Superior de Ingeniería IndustrialUniversidad Politécnica de Cartagen