An ensemble approach of dual base learners for multi-class classification problems

In this work, we formalise and evaluate an ensemble of classifiers that is designed for the resolution of multi-class problems. To achieve a good accuracy rate, the base learners are built with pairwise coupled binary and multi-class classifiers. Moreover, to reduce the computational cost of the ensemble and to improve its performance, these classifiers are trained using a specific attribute subset. This proposal offers the opportunity to capture the advantages provided by binary decomposition methods, by attribute partitioning methods, and by cooperative characteristics associated with a combination of redundant base learners. To analyse the quality of this architecture, its performance has been tested on different domains, and the results have been compared to other well-known classification methods. This experimental evaluation indicates that our model is, in most cases, as accurate as these methods, but it is much more efficient. (C) 2014 Elsevier B.V. All rights reserved.This research was supported by the Spanish MICINN under Projects TRA2010-20225-C03-01, TRA 2011-29454-C03-02, and TRA 2011-29454-C03-03

Comparing combination rules of pairwise neural networks classifiers

Abstract. A decomposition approach to multiclass classification problems consists in decomposing a multiclass problem into a set of binary ones. Decomposition splits the complete multiclass problem into a set of smaller classification problems involving only two classes (binary classification: dichotomies). With a decomposition, one has to define a recombination which recomposes the outputs of the dichotomizers in order to solve the original multiclass problem. There are several approaches to the decomposition, the most famous ones being one-againt-all and one-against-one also called pairwise. In this paper, we focus on pairwise decomposition approach to multiclass classification with neural networks as the base learner for the dichotomies. We are primarily interested in the different possible ways to perform the so-called recombination (or decoding). We review standard methods used to decode the decomposition generated by a one-against-one approach. New decoding methods are proposed and compared to standard methods. A stacking decoding is also proposed which consists in replacing the whole decoding or a part of it by a trainable classifier to arbiter among the conflicting predictions of the pairwise classifiers. Proposed methods try to cope with the main problem while using pairwise decomposition: the use of irrelevant classifiers. Substantial gain is obtained on all datasets used in the experiments. Based on the above, we provide future research directions which consider the recombination problem as an ensemble method

Diseño. análisis y evaluación de conjuntos de clasificadores basados en redes de neuronas

Una de las áreas de investigación que, dentro del marco del Aprendizaje Automático, más atención ha recibido durante las últimas décadas ha sido el diseño de conjuntos de clasificadores. Bajo este denominador se engloban un gran número de algoritmos cuyo objetivo es la construcción de un clasificador robusto haciendo uso de clasificadores más simples denominados clasificadores base. Aunque el uso de los conjuntos de clasificadores se puede argumentar desde diversas perspectivas, la justificación más evidente se encuentra en el comportamiento humano. Antes de tomar una decisión importante es habitual pedir opinión a varios expertos para así tener mayor certeza de que la opción elegida es la más adecuada. Diversos estudios han demostrado que el éxito de cualquier conjunto de clasificadores viene determinado por la precisión y la diversidad de los clasificadores que lo integran. En otras palabras, para que un conjunto de clasificadores mejore la precisión de cualquiera de sus miembros se requiere que éstos sean precisos y diversos. Sin embargo, encontrar clasificadores base que, de forma simultánea, satisfagan ambos requisitos no es una tarea fácil. Por ello, en este trabajo se presentan dos nuevas arquitecturas de conjuntos de clasificadores en una de las cuales, sin obviar la diversidad, se fomenta la precisión de los clasificadores base, mientras que en la otra se fomenta la diversidad frente a la precisión. Las diferencias y la complementariedad existente entre ambas arquitecturas permitirá analizar la influencia que, en el comportamiento global del conjunto, tiene la primacía de una de estas particularidades frente a la otra. Aunque, en el mundo real, la mayor parte de los problemas de clasificación engloban a más de dos categorías, muchos de los conjuntos de clasificadores propuestos en la Bibliografía fueron originalmente concebidos para resolver problemas dicotómicos. En ocasiones, el algoritmo que rige el comportamiento de estos modelos puede extrapolarse a problemas multiclase. Sin embargo, en otros muchos casos, el problema multiclase sólo se puede resolver descomponiendo el problema original en subproblemas binarios. Además, la mayor parte de los modelos propuestos, han sido evaluados sobre dominios artificiales en los que el número de atributos con los que se describen los ejemplos es relativamente pequeño. A pesar de esta tendencia, existen un gran número de dominios reales en los que los ejemplos están descritos por cientos o incluso miles de características. La necesidad de disponer de nuevos métodos de clasificación capaces de resolver problemas reales marca uno de los objetivos de esta Tesis Doctoral. Así, las arquitecturas que se proponen en este trabajo han sido concebidas explícitamente para la resolución de problemas en los que el número de categorías es finito y superior a dos y en los que los ejemplos están descritos por un elevado número de atributos. Partiendo de estas dos singularidades, se pretende acotar, en la medida de lo posible, la complejidad y el coste computacional inherentes a la resolución de este tipo de problemas. La viabilidad de las arquitecturas propuestas se ha determinado experimentalmente. Así, el estudio realizado contempla un exhaustivo análisis en el que, sobre distintos dominios, se analiza el comportamiento de las arquitecturas propuestas y se compara con el logrado por algunos de los modelos de clasificación más referenciados en la Bibliografía. -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------The design of Ensemble of Classifiers has been one of the most active research areas in the field of Machine Learning for the last decades. In this area, many different algorithms have been proposed in order to create a more robust classifier that consists of simpler classifiers named base classifiers. Although the use of ensemble of classifiers can be justified by many different reasons, the most obvious justification is related with human decision making process. Before making a decision, it is common to ask several experts to be sure that the chosen option is the optimal. Many studies have demonstrated that the success of any ensemble of classifiers is related to the accuracy and diversity of the different base classifiers of the ensemble. In other words, an ensemble of classifiers could improve the accuracy of any of its individual members if they are accurate and diverse. However, obtaining base classifiers which satisfy both requirements simultaneously is not an easy task. For this reason, this work presents two new ensembles of classifiers: One of these ensembles prioritizes the accuracy of the base classifiers (taking also into account the diversity) and the other promotes diversity over accuracy. These ensembles are different but complement each other, so it will be possible to analyze the influence of these requirements over the global performance of the ensemble. The number of applications that require multiclass categorization is huge in the real world. However, many of the studies related to supervised learning are focused on the resolution of binary problems. Some machine learning algorithms can then be naturally extended to handle the multiclass case. For other algorithms, a direct extension to the multiclass case may be problematic. Typically, in such cases, the multiclass problem is reduced to multiple binary classification problems that can be solved separately. In addition, most of these models have been evaluated in artificial domains in which the number of features used to describe the examples is relatively small. Despite this, there are many real domains in which the examples are described by hundreds or even thousands of features. For this reason, one of the goals of this thesis is the creation of new classification methods for real world. Thus, the ensembles proposed in this work have been designed to be applicable to real domains in which each example is labeled with one of several categories and is described by a large number of features. Taking these characteristics into account, the computational complexity and cost of the proposed methods need to be reduced as much as possible. The viability of the proposed ensembles has been proved empirically. Thus, this thesis makes a comprehensive analysis in which, taking into account different domains, the performance of the proposed ensembles is analyzed and compared with other wellknown classification methods