A novel online LS-SVM approach for regression and classification

In this paper, a novel online least squares support vector machine approach is proposed for classification and regression problems. Gaussian kernel function is used due to its strong generalization capability. The contribution of the paper is twofold. As the first novelty, all parameters of the SVM including the kernel width parameter σ are trained simultaneously when a new sample arrives. Unscented Kalman filter is adopted to train the parameters since it avoids the sub-optimal solutions caused by linearization in contrast to extended Kalman filter. The second novelty is the variable size moving window by an intelligent update strategy for the support vector set. This provides that SVM model captures the dynamics of data quickly while not letting it become clumsy due to the big amount of useless or out-of-date support vector data. Simultaneous training of the kernel parameter by unscented Kalman filter and intelligent update of support vector set provide significant performance using small amount of support vector data for both classification and system identification application results. © 201

An Online Adaptive Machine Learning Framework for Autonomous Fault Detection

The increasing complexity and autonomy of modern systems, particularly in the aerospace industry, demand robust and adaptive fault detection and health management solutions. The development of a data-driven fault detection system that can adapt to varying conditions and system changes is critical to the performance, safety, and reliability of these systems. This dissertation presents a novel fault detection approach based on the integration of the artificial immune system (AIS) paradigm and Online Support Vector Machines (OSVM). Together, these algorithms create the Artificial Immune System augemented Online Support Vector Machine (AISOSVM). The AISOSVM framework combines the strengths of the AIS and OSVM to create a fault detection system that can effectively identify faults in complex systems while maintaining adaptability. The framework is designed using Model-Based Systems Engineering (MBSE) principles, employing the Capella tool and the Arcadia methodology to develop a structured, integrated approach for the design and deployment of the data-driven fault detection system. A key contribution of this research is the development of a Clonal Selection Algorithm that optimizes the OSVM hyperparameters and the V-Detector algorithm parameters, resulting in a more effective fault detection solution. The integration of the AIS in the training process enables the generation of synthetic abnormal data, mitigating the need for engineers to gather large amounts of failure data, which can be impractical. The AISOSVM also incorporates incremental learning and decremental unlearning for the Online Support Vector Machine, allowing the system to adapt online using lightweight computational processes. This capability significantly improves the efficiency of fault detection systems, eliminating the need for offline retraining and redeployment. Reinforcement Learning (RL) is proposed as a promising future direction for the AISOSVM, as it can help autonomously adapt the system performance in near real-time, further mitigating the need for acquiring large amounts of system data for training, and improving the efficiency of the adaptation process by intelligently selecting the best samples to learn from. The AISOSVM framework was applied to real-world scenarios and platform models, demonstrating its effectiveness and adaptability in various use cases. The combination of the AIS and OSVM, along with the online learning and RL integration, provides a robust and adaptive solution for fault detection and health management in complex autonomous systems. This dissertation presents a significant contribution to the field of fault detection and health management by integrating the artificial immune system paradigm with Online Support Vector Machines, developing a structured, integrated approach for designing and deploying data-driven fault detection systems, and implementing reinforcement learning for online, autonomous adaptation of fault management systems. The AISOSVM framework offers a promising solution to address the challenges of fault detection in complex, autonomous systems, with potential applications in a wide range of industries beyond aerospace

Machine Learning and Integrative Analysis of Biomedical Big Data.

Recent developments in high-throughput technologies have accelerated the accumulation of massive amounts of omics data from multiple sources: genome, epigenome, transcriptome, proteome, metabolome, etc. Traditionally, data from each source (e.g., genome) is analyzed in isolation using statistical and machine learning (ML) methods. Integrative analysis of multi-omics and clinical data is key to new biomedical discoveries and advancements in precision medicine. However, data integration poses new computational challenges as well as exacerbates the ones associated with single-omics studies. Specialized computational approaches are required to effectively and efficiently perform integrative analysis of biomedical data acquired from diverse modalities. In this review, we discuss state-of-the-art ML-based approaches for tackling five specific computational challenges associated with integrative analysis: curse of dimensionality, data heterogeneity, missing data, class imbalance and scalability issues

Weighted Incremental–Decremental Support Vector Machines for concept drift with shifting window

We study the problem of learning the data samples’ distribution as it changes in time. This change, known as concept drift, complicates the task of training a model, as the predictions become less and less accurate. It is known that Support Vector Machines (SVMs) can learn weighted input instances and that they can also be trained online (incremental–decremental learning). Combining these two SVM properties, the open problem is to define an online SVM concept drift model with shifting weighted window. The classic SVM model should be retrained from scratch after each window shift. We introduce the Weighted Incremental–Decremental SVM (WIDSVM), a generalization of the incremental–decremental SVM for shifting windows. WIDSVM is capable of learning from data streams with concept drift, using the weighted shifting window technique. The soft margin constrained optimization problem imposed on the shifting window is reduced to an incremental–decremental SVM. At each window shift, we determine the exact conditions for vector migration during the incremental–decremental process. We perform experiments on artificial and real-world concept drift datasets; they show that the classification accuracy of WIDSVM significantly improves compared to a SVM with no shifting window. The WIDSVM training phase is fast, since it does not retrain from scratch after each window shift

Crowd Scene Analysis in Video Surveillance

There is an increasing interest in crowd scene analysis in video surveillance due to the ubiquitously deployed video surveillance systems in public places with high density of objects amid the increasing concern on public security and safety. A comprehensive crowd scene analysis approach is required to not only be able to recognize crowd events and detect abnormal events, but also update the innate learning model in an online, real-time fashion. To this end, a set of approaches for Crowd Event Recognition (CER) and Abnormal Event Detection (AED) are developed in this thesis. To address the problem of curse of dimensionality, we propose a video manifold learning method for crowd event analysis. A novel feature descriptor is proposed to encode regional optical flow features of video frames, where adaptive quantization and binarization of the feature code are employed to improve the discriminant ability of crowd motion patterns. Using the feature code as input, a linear dimensionality reduction algorithm that preserves both the intrinsic spatial and temporal properties is proposed, where the generated low-dimensional video manifolds are conducted for CER and AED. Moreover, we introduce a framework for AED by integrating a novel incremental and decremental One-Class Support Vector Machine (OCSVM) with a sliding buffer. It not only updates the model in an online fashion with low computational cost, but also adapts to concept drift by discarding obsolete patterns. Furthermore, the framework has been improved by introducing Multiple Incremental and Decremental Learning (MIDL), kernel fusion, and multiple target tracking, which leads to more accurate and faster AED. In addition, we develop a framework for another video content analysis task, i.e., shot boundary detection. Specifically, instead of directly assessing the pairwise difference between consecutive frames over time, we propose to evaluate a divergence measure between two OCSVM classifiers trained on two successive frame sets, which is more robust to noise and gradual transitions such as fade-in and fade-out. To speed up the processing procedure, the two OCSVM classifiers are updated online by the MIDL proposed for AED. Extensive experiments on five benchmark datasets validate the effectiveness and efficiency of our approaches in comparison with the state of the art

Kernel PCA and the Nyström method

This thesis treats kernel PCA and the Nystrom method. We present a novel incre- ¨ mental algorithm for calculation of kernel PCA, which we extend to incremental calculation of the Nystrom approximation. We suggest a new data-dependent ¨ method to select the number of data points to include in the Nystrom subset, ¨ and create a statistical hypothesis test for the same purpose. We further present a cross-validation procedure for kernel PCA to select the number of principal components to retain. Finally, we derive kernel PCA with the Nystrom method ¨ in line with linear PCA and study its statistical accuracy through a confidence bound