Computational Optimizations for Machine Learning

The present book contains the 10 articles finally accepted for publication in the Special Issue “Computational Optimizations for Machine Learning” of the MDPI journal Mathematics, which cover a wide range of topics connected to the theory and applications of machine learning, neural networks and artificial intelligence. These topics include, among others, various types of machine learning classes, such as supervised, unsupervised and reinforcement learning, deep neural networks, convolutional neural networks, GANs, decision trees, linear regression, SVM, K-means clustering, Q-learning, temporal difference, deep adversarial networks and more. It is hoped that the book will be interesting and useful to those developing mathematical algorithms and applications in the domain of artificial intelligence and machine learning as well as for those having the appropriate mathematical background and willing to become familiar with recent advances of machine learning computational optimization mathematics, which has nowadays permeated into almost all sectors of human life and activity

Optimum Feature Selection for Recognizing Objects from Satellite Imagery Using Genetic Algorithm

Object recognition is a research area that aims to associate objects to categories or classes. Usually recognition of object specific geospatial features, as building, tree, mountains, roads, and rivers from high-resolution satellite imagery is a time consuming and expensive problem in the maintenance cycle of a Geographic Information System (GIS). Feature selection is the task of selecting a small subset from original features that can achieve maximum classification accuracy and reduce data dimensionality. This subset of features has some very important benefits like, it reduces computational complexity of learning algorithms, saves time, improve accuracy and the selected features can be insightful for the people involved in problem domain. This makes feature selection as an indispensable task in classification task. In our work, we propose wrapper approach based on Genetic Algorithm (GA) as an optimization algorithm to search the space of all possible subsets related to object geospatial features set for the purpose of recognition. GA is wrapped with three different classifier algorithms namely neural network, k-nearest neighbor and decision tree J48 as subset evaluating mechanism. The GA-ANN, GA-KNN and GA-J48 methods are implemented using the WEKA software on dataset that contains 38 extracted features from satellite images using ENVI software. The proposed wrapper approach incorporated the Correlation Ranking Filter (CRF) for spatial features to remove unimportant features. Results suggest that GA based neural classifiers and using CRF for spatial features are robust and effective in finding optimal subsets of features from large data sets

Ant Colony Optimization

Ant Colony Optimization (ACO) is the best example of how studies aimed at understanding and modeling the behavior of ants and other social insects can provide inspiration for the development of computational algorithms for the solution of difficult mathematical problems. Introduced by Marco Dorigo in his PhD thesis (1992) and initially applied to the travelling salesman problem, the ACO field has experienced a tremendous growth, standing today as an important nature-inspired stochastic metaheuristic for hard optimization problems. This book presents state-of-the-art ACO methods and is divided into two parts: (I) Techniques, which includes parallel implementations, and (II) Applications, where recent contributions of ACO to diverse fields, such as traffic congestion and control, structural optimization, manufacturing, and genomics are presented

Comparison of metaheuristic strategies for peakbin selection in proteomic mass spectrometry data

Mass spectrometry (MS) data provide a promising strategy for biomarker discovery. For this purpose, the detection of relevant peakbins in MS data is currently under intense research. Data from mass spectrometry are challenging to analyze because of their high dimensionality and the generally low number of samples available. To tackle this problem, the scientific community is becoming increasingly interested in applying feature subset selection techniques based on specialized machine learning algorithms. In this paper, we present a performance comparison of some metaheuristics: best first (BF), genetic algorithm (GA), scatter search (SS) and variable neighborhood search (VNS). Up to now, all the algorithms, except for GA, have been first applied to detect relevant peakbins in MS data. All these metaheuristic searches are embedded in two different filter and wrapper schemes coupled with Naive Bayes and SVM classifiers

Incorporating feature ranking and evolutionary methods for the classification of high-dimensional DNA microarray gene expression data

Background: DNA microarray gene expression classification poses a challenging task to the machine learning domain. Typically, the dimensionality of gene expression data sets could go from several thousands to over 10,000 genes. A potential solution to this issue is using feature selection to reduce the dimensionality. Aim The aim of this paper is to investigate how we can use feature quality information to improve the precision of microarray gene expression classification tasks. Method: We propose two evolutionary machine learning models based on the eXtended Classifier System (XCS) and a typical feature selection methodology. The first one, which we call FS-XCS, uses feature selection for feature reduction purposes. The second model is GRD-XCS, which uses feature ranking to bias the rule discovery process of XCS. Results: The results indicate that the use of feature selection/ranking methods is essential for tackling high-dimensional classification tasks, such as microarray gene expression classification. However, the results also suggest that using feature ranking to bias the rule discovery process performs significantly better than using the feature reduction method. In other words, using feature quality information to develop a smarter learning procedure is more efficient than reducing the feature set. Conclusion: Our findings have shown that extracting feature quality information can assist the learning process and improve classification accuracy. On the other hand, relying exclusively on the feature quality information might potentially decrease the classification performance (e.g., using feature reduction). Therefore, we recommend a hybrid approach that uses feature quality information to direct the learning process by highlighting the more informative features, but at the same time not restricting the learning process to explore other features

Temporal Information in Data Science: An Integrated Framework and its Applications

Data science is a well-known buzzword, that is in fact composed of two distinct keywords, i.e., data and science. Data itself is of great importance: each analysis task begins from a set of examples. Based on such a consideration, the present work starts with the analysis of a real case scenario, by considering the development of a data warehouse-based decision support system for an Italian contact center company. Then, relying on the information collected in the developed system, a set of machine learning-based analysis tasks have been developed to answer specific business questions, such as employee work anomaly detection and automatic call classification. Although such initial applications rely on already available algorithms, as we shall see, some clever analysis workflows had also to be developed. Afterwards, continuously driven by real data and real world applications, we turned ourselves to the question of how to handle temporal information within classical decision tree models. Our research brought us the development of J48SS, a decision tree induction algorithm based on Quinlan's C4.5 learner, which is capable of dealing with temporal (e.g., sequential and time series) as well as atemporal (such as numerical and categorical) data during the same execution cycle. The decision tree has been applied into some real world analysis tasks, proving its worthiness. A key characteristic of J48SS is its interpretability, an aspect that we specifically addressed through the study of an evolutionary-based decision tree pruning technique. Next, since a lot of work concerning the management of temporal information has already been done in automated reasoning and formal verification fields, a natural direction in which to proceed was that of investigating how such solutions may be combined with machine learning, following two main tracks. First, we show, through the development of an enriched decision tree capable of encoding temporal information by means of interval temporal logic formulas, how a machine learning algorithm can successfully exploit temporal logic to perform data analysis. Then, we focus on the opposite direction, i.e., that of employing machine learning techniques to generate temporal logic formulas, considering a natural language processing scenario. Finally, as a conclusive development, the architecture of a system is proposed, in which formal methods and machine learning techniques are seamlessly combined to perform anomaly detection and predictive maintenance tasks. Such an integration represents an original, thrilling research direction that may open up new ways of dealing with complex, real-world problems.Data science is a well-known buzzword, that is in fact composed of two distinct keywords, i.e., data and science. Data itself is of great importance: each analysis task begins from a set of examples. Based on such a consideration, the present work starts with the analysis of a real case scenario, by considering the development of a data warehouse-based decision support system for an Italian contact center company. Then, relying on the information collected in the developed system, a set of machine learning-based analysis tasks have been developed to answer specific business questions, such as employee work anomaly detection and automatic call classification. Although such initial applications rely on already available algorithms, as we shall see, some clever analysis workflows had also to be developed. Afterwards, continuously driven by real data and real world applications, we turned ourselves to the question of how to handle temporal information within classical decision tree models. Our research brought us the development of J48SS, a decision tree induction algorithm based on Quinlan's C4.5 learner, which is capable of dealing with temporal (e.g., sequential and time series) as well as atemporal (such as numerical and categorical) data during the same execution cycle. The decision tree has been applied into some real world analysis tasks, proving its worthiness. A key characteristic of J48SS is its interpretability, an aspect that we specifically addressed through the study of an evolutionary-based decision tree pruning technique. Next, since a lot of work concerning the management of temporal information has already been done in automated reasoning and formal verification fields, a natural direction in which to proceed was that of investigating how such solutions may be combined with machine learning, following two main tracks. First, we show, through the development of an enriched decision tree capable of encoding temporal information by means of interval temporal logic formulas, how a machine learning algorithm can successfully exploit temporal logic to perform data analysis. Then, we focus on the opposite direction, i.e., that of employing machine learning techniques to generate temporal logic formulas, considering a natural language processing scenario. Finally, as a conclusive development, the architecture of a system is proposed, in which formal methods and machine learning techniques are seamlessly combined to perform anomaly detection and predictive maintenance tasks. Such an integration represents an original, thrilling research direction that may open up new ways of dealing with complex, real-world problems

Feature selection and modelling methods for microarray data from acute coronary syndrome

Acute coronary syndrome (ACS) represents a leading cause of mortality and morbidity worldwide. Providing better diagnostic solutions and developing therapeutic strategies customized to the individual patient represent societal and economical urgencies. Progressive improvement in diagnosis and treatment procedures require a thorough understanding of the underlying genetic mechanisms of the disease. Recent advances in microarray technologies together with the decreasing costs of the specialized equipment enabled affordable harvesting of time-course gene expression data. The high-dimensional data generated demands for computational tools able to extract the underlying biological knowledge. This thesis is concerned with developing new methods for analysing time-course gene expression data, focused on identifying differentially expressed genes, deconvolving heterogeneous gene expression measurements and inferring dynamic gene regulatory interactions. The main contributions include: a novel multi-stage feature selection method, a new deconvolution approach for estimating cell-type specific signatures and quantifying the contribution of each cell type to the variance of the gene expression patters, a novel approach to identify the cellular sources of differential gene expression, a new approach to model gene expression dynamics using sums of exponentials and a novel method to estimate stable linear dynamical systems from noisy and unequally spaced time series data. The performance of the proposed methods was demonstrated on a time-course dataset consisting of microarray gene expression levels collected from the blood samples of patients with ACS and associated blood count measurements. The results of the feature selection study are of significant biological relevance. For the first time is was reported high diagnostic performance of the ACS subtypes up to three months after hospital admission. The deconvolution study exposed features of within and between groups variation in expression measurements and identified potential cell type markers and cellular sources of differential gene expression. It was shown that the dynamics of post-admission gene expression data can be accurately modelled using sums of exponentials, suggesting that gene expression levels undergo a transient response to the ACS events before returning to equilibrium. The linear dynamical models capturing the gene regulatory interactions exhibit high predictive performance and can serve as platforms for system-level analysis, numerical simulations and intervention studies

A Revision of Procedural Knowledge in the conML Framework

Machine learning methods have been used very successfully for quite some time to recognize patterns, model correlations and generate hypotheses. However, the possibilities for weighing and evaluating the resulting models and hypotheses, and the search for alternatives and contradictions are still predominantly reserved for humans. For this purpose, the novel concept of constructivist machine learning (conML) formalizes limitations of model validity and employs constructivist learning theory to enable doubting of new and existing models with the possibility of integrating, discarding, combining, and abstracting knowledge. The present work identifies issues that impede the systems capability to abstract knowledge from generated models for tasks that lie in the domain of procedural knowledge, and proposes and implements identified solutions. To this end, the conML framework has been reimplemented in the Julia programming language and subsequently been extended. Using a synthetic dataset of impedance spectra of modeled epithelia that has previously been analyzed with an existing implementation of conML, existing and new implementations are tested for consistency and proposed algorithmic changes are evaluated with respect to changes in model generation and abstraction ability when exploring unknown data. Recommendations for specific settings and suggestions for further research are derived from the results. In terms of performance, flexibility and extensibility, the new implementation of conML in Julia provides a good starting point for further research and application of the system.:Contents Abstract . . . . . III Zusammenfassung . . . . . IV Danksagung . . . . . V Selbstständigkeitserklärung . . . . . V 1. Introduction 1.1. Research Questions . . . . . 2 2. Related Work 2.1. Hybrid AI Systems . . . . . 5 2.2. Constructivist Machine Learning (conML) . . . . . 6 2.3. Implemented Methods . . . . . 9 2.3.1. Unsupervised Machine Learning . . . . . 9 2.3.2. Supervised Machine Learning . . . . . 11 2.3.3. Supervised Feature Selection . . . . . 13 2.3.4. Unsupervised Feature Selection . . . . . 17 3. Methods and Implementation 3.1. Notable Algorithmic Changes . . . . . 19 3.1.1. Rescaling of Target Values . . . . . 19 3.1.2. ExtendedWinner Selection . . . . . 21 3.2. Package Structure . . . . . 23 3.3. Interfaces and Implementation of Specific Methods . . . . . 29 3.4. Datasets . . . . . 41 4. Results 4.1. Validation Against the conML Prototype . . . . . 43 4.2. Change in Abstraction Capability . . . . . 49 4.2.1. Influence of Target Scaling . . . . . 49 4.2.2. Influence of the Parameter kappa_p . . . . . 55 4.2.3. Influence of the Winner Selection Procedure . . . . . 61 5. Discussion 5.1. Reproduction Results . . . . . 67 5.2. Rescaling of Constructed Targets . . . . . 69 5.3. kappa_p and the Selection of Winner Models . . . . . 71 6. Conclusions 6.1. Contributions of this Work . . . . . 77 6.2. Future Work . . . . . 78 A. Julia Language Reference . . . . . 81 B. Additional Code Listings . . . . . 91 C. Available Parameters . . . . . 99 C.1. Block Processing . . . . . 105 D. Configurations Reference . . . . . 107 D.1. Unsupervised Methods . . . . . 107 D.2. Supervised Methods . . . . . 108 D.3. Feature Selection . . . . . 109 D.4. Winner Selection . . . . . 110 D.5. General Settings . . . . . 110 E. Supplemental Figures . . . . . 113 E.1. Replacing MAPE with RMSE for Z-Transform Target Scaling . . . . . 113 E.2. Combining Target Rescaling, Winner Selection and High kappa_p . . . . . 119 Bibliography . . . . . 123 List of Figures . . . . . 129 List of Listings . . . . . 133 List of Tables . . . . . 135Maschinelle Lernverfahren werden seit geraumer Zeit sehr erfolgreich zum Erkennen von Mustern, Abbilden von Zusammenhängen und Generieren von Hypothesen eingesetzt. Die Möglichkeiten zum Abwägen und Bewerten der entstandenen Modelle und Hypothesen, und die Suche nach Alternativen und Widersprüchen sind jedoch noch überwiegend dem Menschen vorbehalten. Das neuartige Konzept des konstruktivistischen maschinellen Lernens (conML) formalisiert dazu die Grenzen der Gültigkeit von Modellen und ermöglicht mittels konstruktivistischer Lerntheorie ein Zweifeln über neue und bestehende Modelle mit der Möglichkeit zum Integrieren, Verwerfen, Kombinieren und Abstrahieren von Wissen. Die vorliegende Arbeit identifiziert Probleme, die die Abstraktionsfähigkeit des Systems bei Aufgabenstellungen in der Prozeduralen Wissensdomäne einschränken, bietet Lösungsvorschläge und beschreibt deren Umsetzung. Das algorithmische Framework conML ist dazu in der Programmiersprache Julia reimplementiert und anschließend erweitert worden. Anhand eines synthetischen Datensatzes von Impedanzspektren modellierter Epithelien, der bereits mit einem Prototypen des conML Systems analysiert worden ist, werden bestehende und neue Implementierung auf Konsistenz geprüft und die vorgeschlagenen algorithmischen Änderungen im Hinblick auf Veränderungen beim Erzeugen von Modellen und der Abstraktionsfähigkeit bei der Exploration unbekannter Daten untersucht. Aus den Ergebnissen werden Empfehlungen zu konkreten Einstellungen sowie Vorschläge für weitere Untersuchungen abgeleitet. Die neue Implementierung von conML in Julia bietet im Hinblick auf Performanz, Flexibilität und Erweiterbarkeit einen guten Ausgangspunkt für weitere Forschung und Anwendung des Systems.:Contents Abstract . . . . . III Zusammenfassung . . . . . IV Danksagung . . . . . V Selbstständigkeitserklärung . . . . . V 1. Introduction 1.1. Research Questions . . . . . 2 2. Related Work 2.1. Hybrid AI Systems . . . . . 5 2.2. Constructivist Machine Learning (conML) . . . . . 6 2.3. Implemented Methods . . . . . 9 2.3.1. Unsupervised Machine Learning . . . . . 9 2.3.2. Supervised Machine Learning . . . . . 11 2.3.3. Supervised Feature Selection . . . . . 13 2.3.4. Unsupervised Feature Selection . . . . . 17 3. Methods and Implementation 3.1. Notable Algorithmic Changes . . . . . 19 3.1.1. Rescaling of Target Values . . . . . 19 3.1.2. ExtendedWinner Selection . . . . . 21 3.2. Package Structure . . . . . 23 3.3. Interfaces and Implementation of Specific Methods . . . . . 29 3.4. Datasets . . . . . 41 4. Results 4.1. Validation Against the conML Prototype . . . . . 43 4.2. Change in Abstraction Capability . . . . . 49 4.2.1. Influence of Target Scaling . . . . . 49 4.2.2. Influence of the Parameter kappa_p . . . . . 55 4.2.3. Influence of the Winner Selection Procedure . . . . . 61 5. Discussion 5.1. Reproduction Results . . . . . 67 5.2. Rescaling of Constructed Targets . . . . . 69 5.3. kappa_p and the Selection of Winner Models . . . . . 71 6. Conclusions 6.1. Contributions of this Work . . . . . 77 6.2. Future Work . . . . . 78 A. Julia Language Reference . . . . . 81 B. Additional Code Listings . . . . . 91 C. Available Parameters . . . . . 99 C.1. Block Processing . . . . . 105 D. Configurations Reference . . . . . 107 D.1. Unsupervised Methods . . . . . 107 D.2. Supervised Methods . . . . . 108 D.3. Feature Selection . . . . . 109 D.4. Winner Selection . . . . . 110 D.5. General Settings . . . . . 110 E. Supplemental Figures . . . . . 113 E.1. Replacing MAPE with RMSE for Z-Transform Target Scaling . . . . . 113 E.2. Combining Target Rescaling, Winner Selection and High kappa_p . . . . . 119 Bibliography . . . . . 123 List of Figures . . . . . 129 List of Listings . . . . . 133 List of Tables . . . . . 13