Performance Enhancement of Hyperspectral Semantic Segmentation Leveraging Ensemble Networks

Hyperspectral image (HSI) semantic segmentation is a growing field within computer vision, machine learning, and forestry. Due to the separate nature of these communities, research applying deep learning techniques to ground-type semantic segmentation needs improvement, along with working to bring the research and expectations of these three communities together. Semantic segmentation consists of classifying individual pixels within the image based on the features present. Many issues need to be resolved in HSI semantic segmentation including data preprocessing, feature reduction, semantic segmentation techniques, and adversarial training. In this thesis, we tackle these challenges by employing ensemble methods for HSI semantic segmentation. Deep neural networks (DNNs) for classification tasks have been employed in HSI semantic segmentation with great success. The ensemble method in traditional classification is often used to increase performance, but research into applying it to semantic segmentation in HSIs is relatively new. Instead of using a single network approach to classification, the ensemble method employs multiple networks to improve performance. Research into ensemble methods in HSI has seen increased accuracy, but often has higher computational complexity and relies on expensive preprocessing techniques. To showcase the performance increase the ensemble method has on semantic segmentation, we propose the novel flagship model Clustering Ensemble U-Net (CEU-Net). In CEU-Net we (1) use a bagging ensemble technique to reduce the computational complexity, (2) utilize clustering on class labels as an intelligent method of delineating which data goes to each network, thereby making each sub-network an expert on a particular cluster, and (3) implement with or without patching for better data flexibility. It is shown that CEU-Net outperforms existing hyperspectral semantic segmentation methods, achieving better performance with and without patching compared to baseline models. Semantic segmentation models are vulnerable to adversarial attacks and need adversarial training to counteract them. Adversarial attacks are often intelligent attacks that use the knowledge of a trained classifier to create imperceptible perturbations to hurt classification accuracy. Traditional approaches to adversarial robustness focus on training or retraining a single network on attacked data, however, in the presence of multiple attacks these approaches decrease the performance compared to networks trained individually on each attack. To combat adversarial attacks in HSI semantic segmentation, we propose the Adversarial Discriminator Ensemble Network (ADE-Net) which focuses on attack type detection and adversarial robustness under a unified model to preserve per data-type weight optimally while making the overall network robust. In the proposed method, a discriminator network is used to separate data by attack type into their specific attack-expert ensemble sub-network. The ensemble and discriminator networks are trained together using a unified novel loss function to share information between each network. Our approach allows for the presence of multiple attacks mixed together while also labeling attack types during testing. In this thesis, we experimentally show that ADE-Net outperforms the baseline, which is a single network adversarially trained under a mix of multiple attacks, for popular HSI datasets

Data-Driven Modeling For Decision Support Systems And Treatment Management In Personalized Healthcare

Massive amount of electronic medical records (EMRs) accumulating from patients and populations motivates clinicians and data scientists to collaborate for the advanced analytics to create knowledge that is essential to address the extensive personalized insights needed for patients, clinicians, providers, scientists, and health policy makers. Learning from large and complicated data is using extensively in marketing and commercial enterprises to generate personalized recommendations. Recently the medical research community focuses to take the benefits of big data analytic approaches and moves to personalized (precision) medicine. So, it is a significant period in healthcare and medicine for transferring to a new paradigm. There is a noticeable opportunity to implement a learning health care system and data-driven healthcare to make better medical decisions, better personalized predictions; and more precise discovering of risk factors and their interactions. In this research we focus on data-driven approaches for personalized medicine. We propose a research framework which emphasizes on three main phases: 1) Predictive modeling, 2) Patient subgroup analysis and 3) Treatment recommendation. Our goal is to develop novel methods for each phase and apply them in real-world applications. In the fist phase, we develop a new predictive approach based on feature representation using deep feature learning and word embedding techniques. Our method uses different deep architectures (Stacked autoencoders, Deep belief network and Variational autoencoders) for feature representation in higher-level abstractions to obtain effective and more robust features from EMRs, and then build prediction models on the top of them. Our approach is particularly useful when the unlabeled data is abundant whereas labeled one is scarce. We investigate the performance of representation learning through a supervised approach. We perform our method on different small and large datasets. Finally we provide a comparative study and show that our predictive approach leads to better results in comparison with others. In the second phase, we propose a novel patient subgroup detection method, called Supervised Biclustring (SUBIC) using convex optimization and apply our approach to detect patient subgroups and prioritize risk factors for hypertension (HTN) in a vulnerable demographic subgroup (African-American). Our approach not only finds patient subgroups with guidance of a clinically relevant target variable but also identifies and prioritizes risk factors by pursuing sparsity of the input variables and encouraging similarity among the input variables and between the input and target variables. Finally, in the third phase, we introduce a new survival analysis framework using deep learning and active learning with a novel sampling strategy. First, our approach provides better representation with lower dimensions from clinical features using labeled (time-to-event) and unlabeled (censored) instances and then actively trains the survival model by labeling the censored data using an oracle. As a clinical assistive tool, we propose a simple yet effective treatment recommendation approach based on our survival model. In the experimental study, we apply our approach on SEER-Medicare data related to prostate cancer among African-Americans and white patients. The results indicate that our approach outperforms significantly than baseline models

Exploring the use of neural network-based band selection on hyperspectral imagery to identify informative wavelengths for improving classifier task performance

Hyperspectral imagery is a highly dimensional type of data resulting in high computational costs during analysis. Band selection aims to reduce the original hyperspectral image to a smaller subset that reduces these costs while preserving the maximum amount of spectral information within the data. This thesis explores various types of band selection techniques used in hyperspectral image processing. Modifying Neural network-based techniques and observing the effects on the band selection process due to the change in network architecture or objective are of particular focus in this thesis. Herein, a generalized neural network-based band selection technique is developed and compared to state-of-the-art algorithms that are applied to a unique dataset and the Pavia City Center dataset where the subsequent selected bands are fed into a classifier to gather comparison results

Meta-Transformer: A Unified Framework for Multimodal Learning

Multimodal learning aims to build models that can process and relate information from multiple modalities. Despite years of development in this field, it still remains challenging to design a unified network for processing various modalities ($\textit{e.g.}$ natural language, 2D images, 3D point clouds, audio, video, time series, tabular data) due to the inherent gaps among them. In this work, we propose a framework, named Meta-Transformer, that leverages a $\textbf{frozen}$ encoder to perform multimodal perception without any paired multimodal training data. In Meta-Transformer, the raw input data from various modalities are mapped into a shared token space, allowing a subsequent encoder with frozen parameters to extract high-level semantic features of the input data. Composed of three main components: a unified data tokenizer, a modality-shared encoder, and task-specific heads for downstream tasks, Meta-Transformer is the first framework to perform unified learning across 12 modalities with unpaired data. Experiments on different benchmarks reveal that Meta-Transformer can handle a wide range of tasks including fundamental perception (text, image, point cloud, audio, video), practical application (X-Ray, infrared, hyperspectral, and IMU), and data mining (graph, tabular, and time-series). Meta-Transformer indicates a promising future for developing unified multimodal intelligence with transformers. Code will be available at https://github.com/invictus717/MetaTransformerComment: Project website: https://kxgong.github.io/meta_transformer