On the Decomposition of Polychotomies into Dichotomies

Many important classification problems are \emph{polychotomies}, \emph{i.e.} the data are organized into $K$ classes with $K>2$. Given an unknown function $F : Ømega \to \{1, \dots, K\}$ representing a polychotomy, an algorithm aimed at ``learning'' this polychotomy will produce an approximation of $F$, based on a set of pairs $\{(\mathbf{x}^p, F(\mathbf{x}^p))\}_{p=1}^P$. Although in the wide variety of learning tools, there exist some learning algorithms capable of handling polychotomies, many of the interesting tools were designed by nature for dichotomies ($K=2$). Therefore, many researchers are compelled to use techniques to decompose a polychotomy into a series of dichotomies and thus to apply their favorite algorithms to the resolution of a general problem. A decomposition method based on error-correcting codes has been lately proposed and shown to be very efficient. However, this decomposition is designed only on the basis of $K$ without taking the data into account. In this paper, we explore alternatives to this method, still based on the fruitful idea of error-correcting codes, but where the decomposition is inspired by the data at hand. The efficiency of this approach, both for the simplicity of the model and for the generalization, is illustrated by some numerical experiments

Error-correcting codes and applications to large scale classification systems

Thesis (M. Eng.)--Massachusetts Institute of Technology, Dept. of Electrical Engineering and Computer Science, 2009.Includes bibliographical references (p. 37-39).In this thesis, we study the performance of distributed output coding (DOC) and error-Correcting output coding (ECOC) as potential methods for expanding the class of tractable machine-learning problems. Using distributed output coding, we were able to scale a neural-network-based algorithm to handle nearly 10,000 output classes. In particular, we built a prototype OCR engine for Devanagari and Korean texts based upon distributed output coding. We found that the resulting classifiers performed better than existing algorithms, while maintaining small size. Error-correction, however, was found to be ineffective at increasing the accuracy of the ensemble. For each language, we also tested the feasibility of automatically finding a good codebook. Unfortunately, the results in this direction were primarily negative.by Jeremy Scott Hurwitz.M.Eng

Combining Linear Dichomotizers to Construct Nonlinear Polychotomizers

A polychotomizer which assigns the input to one of $K, K \ge 3$, is constructed using a set of dichotomizers which assign the input to one of two classes. We propose techniques to construct a set of linear dichotomizers whose combined decision forms a nonlinear polychotomizer, to extract structure from data. One way is using error-correcting output codes (ECOC). We propose to incorporate soft weight sharing in training a multilayer perceptron (MLP) to force the second layer weights to a bimodal distribution to be able to interpret them as the decomposition matrix of classes in terms of dichotomizers. This technique can also be used to finetune a set of dichotomizers already generated, for example using ECOC; in such a case, ECOC defines the target values for hidden units in an MLP, facilitating training. Simulation results on eight datasets indicate that compared with a linear one-per-class polychotomizer, pairwise linear dichotomizers and ECOC-based linear dichotomizers, this method generates more accurate classifiers. We also propose and test a method of incremental construction whereby the required number of dichotomizers is determined automatically as opposed to assumed a priori

Support Vector Machine for Multiclass Classification

Support vector machines (SVMs) are primarily designed for 2-class classification problems. Although in several papers it is mentioned that the combination of $K$ SVMs can be used to solve a $K$-class classification problem, such a procedure requires some care. In this paper, the scaling problem of different SVMs is highlighted. Various normalization methods are proposed to cope with this problem and their efficiencies are measured empirically. This simple way of using SVMs to learn a $K$-class classification problem consists in choosing the maximum applied to the outputs of $K$ SVMs solving a \textit{one-per-class} decomposition of the general problem. In the second part of this paper, more sophisticated techniques are suggested. On the one hand, a stacking of the $K$ SVMs with other classification techniques is proposed. On the other end, the \textit{one-per-class} decomposition scheme is replaced by more elaborated schemes based on error-correcting codes. An incremental algorithm for the elaboration of pertinent decomposition schemes is mentioned, which exploits the properties of SVMs for an efficient computation

Applicability and Interpretability of Logical Analysis of Data in Condition Based Maintenance

Résumé Cette thèse étudie l’applicabilité et l’adaptabilité d’une approche d’exploration de données basée sur l’intelligence artificielle proposée dans [Hammer, 1986] et appelée analyse logique de données (LAD) aux applications diagnostiques dans le domaine de la maintenance conditionnelle CBM). La plupart des technologies utilisées à ce jour pour la prise de décision dans la maintenance conditionnelle ont tendance à automatiser le processus de diagnostic, sans offrir aucune connaissance ajoutée qui pourrait être utile à l’opération de maintenance et au personnel de maintenance. Par comparaison à d’autres techniques de prise de décision dans le domaine de la CBM, la LAD possède deux avantages majeurs : (1) il s’agit d’une approche non statistique, donc les données n’ont pas à satisfaire des suppositions statistiques et (2) elle génère des formes interprétables qui pourraient aider à résoudre les problèmes de maintenance. Une étude sur l’application de la LAD dans la maintenance conditionnelle est présentée dans cette recherche dont l’objectif est (1) d’étudier l’applicabilité de la LAD dans des situations différentes qui nécessitent des considérations particulières concernant les types de données d’entrée et les décisions de maintenance, (2) d’adapter la méthode LAD aux exigences particulières qui se posent à partir de ces applications et (3) d’améliorer la méthodologie LAD afin d’augmenter l’exactitude de diagnostic et d’interprétation de résultats. Les aspects innovants de la recherche présentés dans cette thèse sont (1) l’application de la LAD dans la CBM pour la première fois dans des applications qui bénéficient des propriétés uniques de cette technologie et (2) les modifications innovatrices de la méthodologie de la LAD, en particulier dans le domaine de la génération des formes, afin d’améliorer ses performances dans le cadre de la CBM et dans le domaine de classification multiclasses. La recherche menée dans cette thèse a suivi une approche évolutive afin d’atteindre les objectifs énoncés ci-dessus. La LAD a été utilisée et adaptée à trois applications : (1) la détection des composants malveillants (Rogue) dans l’inventaire de pièces de rechange réparables d’une compagnie aérienne commerciale, (2) la détection et l’identification des défauts dans les transformateurs de puissance en utilisant la DGA et (3) la détection des défauts dans les rotors en utilisant des signaux de vibration. Cette recherche conclut que la LAD est une approche de prise de décision prometteuse qui ajoute d’importants avantages à la mise en oeuvre de la CBM dans l’industrie.----------Abstract This thesis studies the applicability and adaptability of a data mining artificial intelligence approach called Logical Analysis of Data (LAD) to diagnostic applications in Condition Based Maintenance (CBM). Most of the technologies used so far for decision support in CBM tend to automate the diagnostic process without offering any added knowledge that could be helpful to the maintenance operation and maintenance personnel. LAD possesses two key advantages over other decision making technologies used in CBM: (1) it is a non-statistical approach; as such no statistical assumptions are required for the input data, and (2) it generates interpretable patterns that could help solve maintenance problems. A study on the implementation of LAD in CBM is presented in this research whose objective are to study the applicability of LAD in different CBM situations requiring special considerations regarding the types of input data and maintenance decisions, adapt the LAD methodology to the particular requirements that arise from these applications, and improve the LAD methodology in line with the above two objectives in order to increase diagnosis accuracy and result interpretability. The novelty of the research presented in this thesis is (1) the application of LAD to CBM for the first time in applications that stand to benefit from the advantages that this technology provides; and (2) the innovative modifications to LAD methodology, particularly in the area of pattern generation, in order to improve its performance within the context of CBM. The research conducted in this thesis followed an evolutionary approach in order to achieve the objectives stated in the Introduction. The research applied LAD in three applications: (1) the detection of Rogue components within the spare part inventory of reparable components in a commercial airline company, (2) the detection and identification of faults in power transformers using DGA, and (3) the detection of faults in rotor bearings using vibration signals. This research concludes that LAD is a promising decision making approach that adds important benefits to the implementation of CBM in the industry

Forecasting financial markets using linear, nonlinear & model combination methods

In this thesis we investigate the question of asset price predictability. The two major themes that we focus on are firstly; whether machine learning and statistical modelling techniques, which impose less restrictive assumptions on asset price dynamics than do classical linear methods, can be used to forecast and trade financial markets to a degree greater than that which traditional asset pricing models would lead us to expect and secondly; to what extent model combination/ensemble strategies can add value in this pursuit. The approaches used include support vector regression (SVR), k-nearest neighbours (KNN), trading rules, linear regression (LR) and the random subspace ensemble method. We investigate these two themes using inherently data-driven models across datasets of sufficient size to render statistically meaningful results in three self-contained contexts. The first piece of empirical work compares the relative forecasting performance of SVR, KNN and LR models when applied to predicting daily returns of 58 UK stocks in the FTSE 100 over 4000 days. Bootstrap simulations are used to shed further statistical light on model performance. Secondly, we investigate the extent to which model combinations can improve forecasting performance with the use of the random subspace ensemble method for constructing ensembles of linear regression models to predict the returns of a portfolio of FTSE 100 stocks. The primary ensemble consists of 62500 component models estimated by randomly sampling subsets of the feature set and the final result combined via a majority vote. Lastly, we conduct an in-depth study of the channel break-out trading rule over a portfolio of 37 futures markets. We borrow a page from the book of modern portfolio theory where it is the performance of individual markets in the context of a portfolio that is ultimately of interest rather than on an individual basis. This approach is rarely used in the literature but is able to shed more light on the question of trading rule efficacy. Bootstrap resampling is employed to derive robust performance statistics. Our results show the Sharpe Ratio of the portfolio to be three times greater than of individual markets as a result of diversification in addition to being greater than that of S&P500 benchmark. We did not set out in an attempt to refute the weak form of Fama's (1970) classic taxonomy of information sets or, colloquially, "to beat the market"; nonetheless, some of our results suggest economically significant returns

Diagnosis of Machining Conditions Based on Logical Analysis of Data

RÉSUMÉ : Un élément clé pour un système d'usinage automatisé sans surveillance est le développement de systèmes de surveillance et de contrôle fiables et robustes. Plusieurs modèles mathématiques et statistiques, qui modélisent la relation entre les variables indépendantes et les variables dépendantes d’usinage, sont suggérés dans la littérature, en commençant par le modèle de Taylor jusqu’aux modèles de régression les plus sophistiqués. Tous ces modèles ne sont pas dynamiques, dans le sens que leurs paramètres ne changent pas avec le temps. Des modèles basés sur l'intelligence artificielle ont résolu de nombreux problèmes dans ce domaine, mais la recherche continue. Dans la présente thèse, je propose l'application d'une approche appelée Analyse Logique de Données (LAD) pour prédire le sortant d’un processus d’usinage. Cette approche a démontré une bonne performance et des capacités additionnelles une fois comparée à la conception traditionnelle des expériences ou à la modélisation mathématique et statistique. Elle est aussi comparée dans cette thèse à la méthode bien connue des réseaux de neurones. Elle est basée sur l'exploitation des données saisies par des capteurs et l'extraction des informations utiles à partir de ces dernières. LAD est utilisé pour déterminer les meilleures conditions d'usinage, pour détecter l'usure de l'outil, pour identifier le moment optimal de remplacement de l’outil d’usinage, et pour surveiller et contrôler les processus d'usinage. Étant donné que les capteurs et les technologies de l'information sont tous les deux en expansion rapide et continue, il serait prévu qu'un outil d’analyse tels que LAD aidera à tracer un chemin dans l'amelioration des processus d'usinage en utilisant les techniques de pointe afin de réduire considérablement le coût ces processus. Les résultats de mon travail pourraient avoir un impact important sur l'optimisation de ces processus.----------ABSTRACT : A key issue for an unattended and automated machining system is the development of reliable and robust monitoring and controlling systems. Research in Artificial Intelligence-based monitoring of machining systems covers several issues and has solved many problems, but the search continues for a robust technique that does not depend on a statistical learning background and that does not have ambiguous procedures. In this thesis, I propose the application of an approach called Logical Analysis of Data (LAD) which is based on the exploitation of data captured by sensors, and the extraction of useful information from this data. LAD is used for determining the best machining conditions, detecting the tool wear, identifying the optimal replacement time for machining tools, monitoring, and controlling machining processes. LAD has demonstrated good performance and additional capabilities when it is compared to the famous statistical technique, Proportional Hazard Model (PHM), and the well known machine learning technique, Artificial Neural Network (ANN). Since sensors’ and information technologies are both expanding rapidly and continuously, it is expected that an analysis tool such as LAD will help in blazing a new trail in machining processes by using state of the art techniques in order to significantly reduce the cost of machining process

Fault Prognostics Using Logical Analysis of Data and Non-Parametric Reliability Estimation Methods

RÉSUMÉ : Estimer la durée de vie utile restante (RUL) d’un système qui fonctionne suivant différentes conditions de fonctionnement représente un grand défi pour les chercheurs en maintenance conditionnelle (CBM). En effet, il est difficile de comprendre la relation entre les variables qui représentent ces conditions de fonctionnement et la RUL dans beaucoup de cas en pratique à cause du degré élevé de corrélation entre ces variables et leur dépendance dans le temps. Il est également difficile, voire impossible, pour des experts d’acquérir et accumuler un savoir à propos de systèmes complexes, où l'échec de l'ensemble du système est vu comme le résultat de l'interaction et de la concurrence entre plusieurs modes de défaillance. Cette thèse présente des méthodologies pour le pronostic en CBM basé sur l'apprentissage automatique, et une approche de découverte de connaissances appelée Logical Analysis of Data (LAD). Les méthodologies proposées se composent de plusieurs implémentations de la LAD combinées avec des méthodes non paramétriques d'estimation de fiabilité. L'objectif de ces méthodologies est de prédire la RUL du système surveillé tout en tenant compte de l'analyse des modes de défaillance uniques ou multiples. Deux d’entre elles considèrent un mode de défaillance unique et une autre considère de multiples modes de défaillance. Les deux méthodologies pour le pronostic avec mode unique diffèrent dans la manière de manipuler les données. Les méthodologies de pronostique dans cette recherche doctorale ont été testées et validées sur la base d'un ensemble de tests bien connus. Dans ces tests, les méthodologies ont été comparées à des techniques de pronostic connues; le modèle à risques proportionnels de Cox (PHM), les réseaux de neurones artificiels (ANNs) et les machines à vecteurs de support (SVMs). Deux ensembles de données ont été utilisés pour illustrer la performance des trois méthodologies: l'ensemble de données du turboréacteur à double flux (turbofan) qui est disponible au sein de la base de données pour le développement d'algorithmes de pronostic de la NASA, et un autre ensemble de données obtenu d’une véritable application dans l'industrie. Les résultats de ces comparaisons indiquent que chacune des méthodologies proposées permet de prédire avec précision la RUL du système considéré. Cette recherche doctorale conclut que l’approche utilisant la LAD possède d’importants mérites et avantages qui pourraient être bénéfiques au domaine du pronostic en CBM. Elle est capable de gérer les données en CBM qui sont corrélées et variantes dans le temps. Son autre avantage et qu’elle génère un savoir interprétable qui est bénéfique au personnel de maintenance.----------ABSTRACT : Estimating the remaining useful life (RUL) for a system working under different operating conditions represents a big challenge to the researchers in the condition-based maintenance (CBM) domain. The reason is that the relationship between the covariates that represent those operating conditions and the RUL is not fully understood in many practical cases, due to the high degree of correlation between such covariates, and their dependence on time. It is also difficult or even impossible for the experts to acquire and accumulate the knowledge from a complex system, where the failure of the system is regarded as the result of interaction and competition between several failure modes. This thesis presents systematic CBM prognostic methodologies based on a pattern-based machine learning and knowledge discovery approach called Logical Analysis of Data (LAD). The proposed methodologies comprise different implementations of the LAD approach combined with non-parametric reliability estimation methods. The objective of these methodologies is to predict the RUL of the monitored system while considering the analysis of single or multiple failure modes. Three different methodologies are presented; two deal with single failure mode and one deals with multiple failure modes. The two methodologies for single mode prognostics differ in the way of representing the data. The prognostic methodologies in this doctoral research have been tested and validated based on a set of widely known tests. In these tests, the methodologies were compared to well-known prognostic techniques; the proportional hazards model (PHM), artificial neural networks (ANNs) and support vector machines (SVMs). Two datasets were used to illustrate the performance of the three methodologies: the turbofan engine dataset that is available at NASA prognostic data repository, and another dataset collected from a real application in the industry. The results of these comparisons indicate that each of the proposed methodologies provides an accurate prediction for the RUL of the monitored system. This doctoral research concludes that the LAD approach has attractive merits and advantages that add benefits to the field of prognostics. It is capable of dealing with the CBM data that are correlated and time-varying. Another advantage is its generation of an interpretable knowledge that is beneficial to the maintenance personnel

On the Decomposition of Polychotomies Into Dichotomies

Many important classification problems are polychotomies, i.e. the data are organized into K classes with K ? 2. Given an unknown function F :\Omega ! f1; : : : ; Kg representing a polychotomy, an algorithm aimed at "learning" this polychotomy will produce an approximation of F , based on the knowledge of a set of pairs f(x p ; F (x p ))g P p=1 . Although in the wide variety of learning tools there exist some learning algorithms capable of handling polychotomies, many of the interesting tools were designed by nature for dichotomies (K = 2). Therefore, many researchers are compelled to use techniques to decompose a polychotomy into a series of dichotomies in order to apply their favorite algorithms to the resolution of a general problem. A decomposition method based on error-correcting codes has been lately proposed and shown to be very efficient. However, this decomposition is designed only on the basis of K without taking the data into account. In this paper, we explore alter..

Parallel Non-Linear Dichotomizers

We present a new learning machine model for classication problems, based on decompositions of multiclass classication problems in sets of two-class subproblems, assigned to non linear dichotomizers that learn their task independently of each other. The experimentation performed on classical data sets, shows that this learning machine model achieves signicant performance improvements over MLP, and previous classiers models based on decomposition of polychotomies into dichotomies. The theoretical reasons of the good properties of generalization of the proposed learning machine model are explained in the framework of the statistical learning theory. Keywords: Generalization, learning machines for classication, decomposition of polychotomies into dichotomies, statistical learning theory. 1 Introduction Several learning methods implementing inductive principles of empirical risk minimization [4], regularization [9], structural risk minimization [23], bayesian inference [7], mi..