Classifier selection with permutation tests

This work presents a content-based recommender system for machine learning classifier algorithms. Given a new data set, a recommendation of what classifier is likely to perform best is made based on classifier performance over similar known data sets. This similarity is measured according to a data set characterization that includes several state-of-the-art metrics taking into account physical structure, statistics, and information theory. A novelty with respect to prior work is the use of a robust approach based on permutation tests to directly assess whether a given learning algorithm is able to exploit the attributes in a data set to predict class labels, and compare it to the more commonly used F-score metric for evaluating classifier performance. To evaluate our approach, we have conducted an extensive experimentation including 8 of the main machine learning classification methods with varying configurations and 65 binary data sets, leading to over 2331 experiments. Our results show that using the information from the permutation test clearly improves the quality of the recommendations.Peer ReviewedPostprint (author's final draft

Detección de fallas en cajas de engranajes utilizando el método de aprendizaje de máquinas Support Vector Machine (SVM)

El objetivo de esta investigación fue crear un modelo predictivo bajo el enfoque de aprendizaje de máquinas y verificar su efectividad para clasificar y detectar fallas en cajas de engranajes de manera automática, para lo cual se utilizó un conjunto de datos de señales de vibración obtenido del repositorio de Iniciativa de Datos de Energía Abierta (OEDI) del departamento de energía de EE. UU. La creación del modelo se llevó a cabo utilizando el método de aprendizaje de máquinas supervisado Support Vector Machine (SVM) y con la ayuda del software de programación Python, donde se realizó el preprocesamiento y análisis del conjunto de datos. Al conjunto de datos se le extrajo características en el dominio del tiempo y dominio de la frecuencia. Para seleccionar las mejores características se aplicó el método de Eliminación Recursiva de Características con Validación Cruzada (RFECV). Para ingresar al clasificador SVM los datos se dividieron en 70% para entrenamiento y 30% para prueba. Como resultado se obtuvo tres modelos de detección de fallas, un primer modelo donde se utilizó un conjunto de datos recopilados por cuatro acelerómetros bajo una carga de 50%, un segundo modelo donde se combinó los datos recopilados por cuatro acelerómetros y cargas en un rango de 0 a 90% y un tercer modelo utilizando los datos de un solo acelerómetro del modelo dos. Cada modelo se entrenó y probo obteniéndose excelentes resultados, logrando una exactitud de 99,84% y una precisión de 99,82% para el mejor modelo. Los resultados demuestran que el método empleado clasifica y predice fallas con alta exactitud y precisión, siendo un método prometedor y de gran aporte para el mantenimiento industrial. Se recomienda reducir y estandarizar el conjunto de características, de esa forma se consigue reducir la carga computacional y a su vez mejorar el rendimiento del modelo.The objective of this research was to create a predictive model under the machine learning approach and verify its effectiveness to classify and detect faults in gearboxes automatically, for which a data set of vibration signals obtained from the repository was used from the Open Energy Data Initiative (OEDI) of the US Department of Energy. The creation of the model was carried out using the Support Vector Machine (SVM) supervised machine learning method and with the aid of Python programming software, where the preprocessing and analysis of the data set was performed. Features in the time domain and frequency domain were extracted from the data set. To select the best features, the Recursive Features Elimination with Cross Validation (RFECV) method was applied. To enter the SVM classifier, the data was divided into 70% for training and 30% for testing. As a result, three fault detection models were obtained, a first model where a set of data collected by four accelerometers under a load of 50% was produced, a second model where the data collected by four accelerometers and loads in a range of 0 to 90% and a third model using the data from a single accelerometer of model two. Each model was trained and tested obtaining excellent results, achieving an accuracy of 99,84% and a precision of 99,82% for the best model. The results show that the method used classifies and predicts faults with high accuracy and precision, being a promising method and of great contribution to industrial maintenance. It is recommended to reduce and standardize the set of features, in this way it is possible to reduce the computational load and in turn improve the performance of the model

Classifier selection with permutation tests

This work presents a content-based recommender system for machine learning classifier algorithms. Given a new data set, a recommendation of what classifier is likely to perform best is made based on classifier performance over similar known data sets. This similarity is measured according to a data set characterization that includes several state-of-the-art metrics taking into account physical structure, statistics, and information theory. A novelty with respect to prior work is the use of a robust approach based on permutation tests to directly assess whether a given learning algorithm is able to exploit the attributes in a data set to predict class labels, and compare it to the more commonly used F-score metric for evaluating classifier performance. To evaluate our approach, we have conducted an extensive experimentation including 8 of the main machine learning classification methods with varying configurations and 65 binary data sets, leading to over 2331 experiments. Our results show that using the information from the permutation test clearly improves the quality of the recommendations.Peer Reviewe