An Online Adaptive Machine Learning Framework for Autonomous Fault Detection

The increasing complexity and autonomy of modern systems, particularly in the aerospace industry, demand robust and adaptive fault detection and health management solutions. The development of a data-driven fault detection system that can adapt to varying conditions and system changes is critical to the performance, safety, and reliability of these systems. This dissertation presents a novel fault detection approach based on the integration of the artificial immune system (AIS) paradigm and Online Support Vector Machines (OSVM). Together, these algorithms create the Artificial Immune System augemented Online Support Vector Machine (AISOSVM). The AISOSVM framework combines the strengths of the AIS and OSVM to create a fault detection system that can effectively identify faults in complex systems while maintaining adaptability. The framework is designed using Model-Based Systems Engineering (MBSE) principles, employing the Capella tool and the Arcadia methodology to develop a structured, integrated approach for the design and deployment of the data-driven fault detection system. A key contribution of this research is the development of a Clonal Selection Algorithm that optimizes the OSVM hyperparameters and the V-Detector algorithm parameters, resulting in a more effective fault detection solution. The integration of the AIS in the training process enables the generation of synthetic abnormal data, mitigating the need for engineers to gather large amounts of failure data, which can be impractical. The AISOSVM also incorporates incremental learning and decremental unlearning for the Online Support Vector Machine, allowing the system to adapt online using lightweight computational processes. This capability significantly improves the efficiency of fault detection systems, eliminating the need for offline retraining and redeployment. Reinforcement Learning (RL) is proposed as a promising future direction for the AISOSVM, as it can help autonomously adapt the system performance in near real-time, further mitigating the need for acquiring large amounts of system data for training, and improving the efficiency of the adaptation process by intelligently selecting the best samples to learn from. The AISOSVM framework was applied to real-world scenarios and platform models, demonstrating its effectiveness and adaptability in various use cases. The combination of the AIS and OSVM, along with the online learning and RL integration, provides a robust and adaptive solution for fault detection and health management in complex autonomous systems. This dissertation presents a significant contribution to the field of fault detection and health management by integrating the artificial immune system paradigm with Online Support Vector Machines, developing a structured, integrated approach for designing and deploying data-driven fault detection systems, and implementing reinforcement learning for online, autonomous adaptation of fault management systems. The AISOSVM framework offers a promising solution to address the challenges of fault detection in complex, autonomous systems, with potential applications in a wide range of industries beyond aerospace

Efficient cross-validation for kernelized least-squares regression with sparse basis expansions

We propose an efficient algorithm for calculating hold-out and cross-validation (CV) type of estimates for sparse regularized least-squares predictors. Holding out H data points with our method requires O(min(H^2n,Hn^2)) time provided that a predictor with n basis vectors is already trained. In addition to holding out training examples, also some of the basis vectors used to train the sparse regularized least-squares predictor with the whole training set can be removed from the basis vector set used in the hold-out computation. In our experiments, we demonstrate the speed improvements provided by our algorithm in practice, and we empirically show the benefits of removing some of the basis vectors during the CV rounds

Functional-bandwidth kernel for Support Vector Machine with Functional Data:An alternating optimization algorithm

Functional Data Analysis (FDA) is devoted to the study of data which are functions. Support Vector Ma- chine (SVM) is a benchmark tool for classification, in particular, of functional data. SVM is frequently used with a kernel (e.g.: Gaussian) which involves a scalar bandwidth parameter. In this paper, we pro- pose to use kernels with functional bandwidths. In this way, accuracy may be improved, and the time intervals critical for classification are identified. Tuning the functional parameters of the new kernel is a challenging task expressed as a continuous optimization problem, solved by means of a heuristic. Our experiments with benchmark data sets show the advantages of using functional parameters and the ef- fectiveness of our approach

Understanding and Adapting Tree Ensembles: A Training Data Perspective

Despite the impressive success of deep-learning models on unstructured data (e.g., images, audio, text), tree-based ensembles such as random forests and gradient-boosted trees are hugely popular and remain the preferred choice for tabular or structured data, and are regularly used to win challenges on data-competition websites such as Kaggle and DrivenData. Despite their impressive predictive performance, tree-based ensembles lack certain characteristics which may limit their further adoption, especially for safety-critical or privacy-sensitive domains such as weather forecasting or predictive medical modeling. This dissertation investigates the shortcomings currently facing tree-based ensembles---lack of explainable predictions, limited uncertainty estimation, and inefficient adaptability to changes in the training data---and posits that numerous improvements to tree-based ensembles can be made by analyzing the relationships between the training data and the resulting learned model. By studying the effects of one or many training examples on tree-based ensembles, we develop solutions for these models which (1) increase their predictive explainability, (2) provide accurate uncertainty estimates for individual predictions, and (3) efficiently adapt learned models to accurately reflect updated training data. This dissertation includes previously published coauthored material

Constructing Prediction Intervals with Neural Networks: An Empirical Evaluation of Bootstrapping and Conformal Inference Methods

Artificial neural networks (ANNs) are popular tools for accomplishing many machine learning tasks, including predicting continuous outcomes. However, the general lack of confidence measures provided with ANN predictions limit their applicability, especially in military settings where accuracy is paramount. Supplementing point predictions with prediction intervals (PIs) is common for other learning algorithms, but the complex structure and training of ANNs renders constructing PIs difficult. This work provides the network design choices and inferential methods for creating better performing PIs with ANNs to enable their adaptation for military use. A two-step experiment is executed across 11 datasets, including an imaged-based dataset. Two non-parametric methods for constructing PIs, bootstrapping and conformal inference, are considered. The results of the first experimental step reveal that the choices inherent to building an ANN affect PI performance. Guidance is provided for optimizing PI performance with respect to each network feature and PI method. In the second step, 20 algorithms for constructing PIs—each using the principles of bootstrapping or conformal inference—are implemented to determine which provides the best performance while maintaining reasonable computational burden. In general, this trade-off is optimized when implementing the cross-conformal method, which maintained interval coverage and efficiency with decreased computational burden

Dynamic Data Mining: Methodology and Algorithms

Supervised data stream mining has become an important and challenging data mining task in modern organizations. The key challenges are threefold: (1) a possibly infinite number of streaming examples and time-critical analysis constraints; (2) concept drift; and (3) skewed data distributions. To address these three challenges, this thesis proposes the novel dynamic data mining (DDM) methodology by effectively applying supervised ensemble models to data stream mining. DDM can be loosely defined as categorization-organization-selection of supervised ensemble models. It is inspired by the idea that although the underlying concepts in a data stream are time-varying, their distinctions can be identified. Therefore, the models trained on the distinct concepts can be dynamically selected in order to classify incoming examples of similar concepts. First, following the general paradigm of DDM, we examine the different concept-drifting stream mining scenarios and propose corresponding effective and efficient data mining algorithms. • To address concept drift caused merely by changes of variable distributions, which we term pseudo concept drift, base models built on categorized streaming data are organized and selected in line with their corresponding variable distribution characteristics. • To address concept drift caused by changes of variable and class joint distributions, which we term true concept drift, an effective data categorization scheme is introduced. A group of working models is dynamically organized and selected for reacting to the drifting concept. Secondly, we introduce an integration stream mining framework, enabling the paradigm advocated by DDM to be widely applicable for other stream mining problems. Therefore, we are able to introduce easily six effective algorithms for mining data streams with skewed class distributions. In addition, we also introduce a new ensemble model approach for batch learning, following the same methodology. Both theoretical and empirical studies demonstrate its effectiveness. Future work would be targeted at improving the effectiveness and efficiency of the proposed algorithms. Meantime, we would explore the possibilities of using the integration framework to solve other open stream mining research problems

Object detection for big data

"May 2014."Dissertation supervisor: Dr. Tony X. Han.Includes vita.We have observed significant advances in object detection over the past few decades and gladly seen the related research has began to contribute to the world: Vehicles could automatically stop before hitting any pedestrian; Face detectors have been integrated into smart phones and tablets; Video surveillance systems could locate the suspects and stop crimes. All these applications demonstrate the substantial research progress on object detection. However learning a robust object detector is still quite challenging due to the fact that object detection is a very unbalanced big data problem. In this dissertation, we aim at improving the object detector's performance from different aspects. For object detection, the state-of-the-art performance is achieved through supervised learning. The performances of object detectors of this kind are mainly determined by two factors: features and underlying classification algorithms. We have done thorough research on both of these factors. Our contribution involves model adaption, local learning, contextual boosting, template learning and feature development. Since the object detection is an unbalanced problem, in which positive examples are hard to be collected, we propose to adapt a general object detector for a specific scenario with a few positive examples; To handle the large intra-class variation problem lying in object detection task, we propose a local adaptation method to learn a set of efficient and effective detectors for a single object category; To extract the effective context from the huge amount of negative data in object detection, we introduce a novel contextual descriptor to iteratively improve the detector; To detect object with a depth sensor, we design an effective depth descriptor; To distinguish the object categories with the similar appearance, we propose a local feature embedding and template selection algorithm, which has been successfully incorporated into a real-world fine-grained object recognition application. All the proposed algorithms and featuIncludes bibliographical references (pages 117-130)