On the class overlap problem in imbalanced data classification.

Class imbalance is an active research area in the machine learning community. However, existing and recent literature showed that class overlap had a higher negative impact on the performance of learning algorithms. This paper provides detailed critical discussion and objective evaluation of class overlap in the context of imbalanced data and its impact on classification accuracy. First, we present a thorough experimental comparison of class overlap and class imbalance. Unlike previous work, our experiment was carried out on the full scale of class overlap and an extreme range of class imbalance degrees. Second, we provide an in-depth critical technical review of existing approaches to handle imbalanced datasets. Existing solutions from selective literature are critically reviewed and categorised as class distribution-based and class overlap-based methods. Emerging techniques and the latest development in this area are also discussed in detail. Experimental results in this paper are consistent with existing literature and show clearly that the performance of the learning algorithm deteriorates across varying degrees of class overlap whereas class imbalance does not always have an effect. The review emphasises the need for further research towards handling class overlap in imbalanced datasets to effectively improve learning algorithms’ performance

Understanding the apparent superiority of over-sampling through an analysis of local information for class-imbalanced data

Data plays a key role in the design of expert and intelligent systems and therefore, data preprocessing appears to be a critical step to produce high-quality data and build accurate machine learning models. Over the past decades, increasing attention has been paid towards the issue of class imbalance and this is now a research hotspot in a variety of fields. Although the resampling methods, either by under-sampling the majority class or by over-sampling the minority class, stand among the most powerful techniques to face this problem, their strengths and weaknesses have typically been discussed based only on the class imbalance ratio. However, several questions remain open and need further exploration. For instance, the subtle differences in performance between the over- and under-sampling algorithms are still under-comprehended, and we hypothesize that they could be better explained by analyzing the inner structure of the data sets. Consequently, this paper attempts to investigate and illustrate the effects of the resampling methods on the inner structure of a data set by exploiting local neighborhood information, identifying the sample types in both classes and analyzing their distribution in each resampled set. Experimental results indicate that the resampling methods that produce the highest proportion of safe samples and the lowest proportion of unsafe samples correspond to those with the highest overall performance. The significance of this paper lies in the fact that our findings may contribute to gain a better understanding of how these techniques perform on class-imbalanced data and why over-sampling has been reported to be usually more efficient than under-sampling. The outcomes in this study may have impact on both research and practice in the design of expert and intelligent systems since a priori knowledge about the internal structure of the imbalanced data sets could be incorporated to the learning algorithms

딥 스파이킹 뉴럴 네트워크의 빠르고 정확한 정보 전달

학위논문 (박사) -- 서울대학교 대학원 : 공과대학 전기·컴퓨터공학부, 2021. 2. 윤성로.오늘 날 딥러닝의 큰 성공은 고성능 병렬 컴퓨팅 시스템의 발전과 복잡한 모델을 학습하기 위해 필요한 많은 양의 데이터가 수집되어 접근이 가능해진 점이라고 할 수 있다. 하지만 실제 세상에 존재하는 더 어려운 문제들을 풀고자할 때는 더욱 더 섬세하고 복잡한 모델과 이 모델을 성공적으로 학습할 수 있는 방대한 양의 데이터를 필요한다. 하지만 이러한 점들은 모델 수행 시 연산 오버헤드와 전력 소모를 급격하게 증가시킬 수 밖에 없다. 이러한 문제점들을 극복하는 여러 방법들 중 하나로 스파이킹 뉴럴 네트워크가 최근 많은 주목을 받고 있다. 스파이킹 뉴럴 네트워크는 제 3세대 인공 신경망으로 불리며 이벤트 중심의 동작을 기반으로 하여 저전력이 가장 큰 장점이다. 스파이킹 뉴럴 네트워크는 실제 인간의 뇌에서 뉴런들 간 정보를 전달하는 방식을 모방하며 스파이킹 뉴런을 연산 단위로 사용하고 있다. 스파이킹 뉴럴 네트워크는 생물학적 신경계와 동일하게 시간적 정보를 활용할 수 있기 때문에 매우 뛰어난 연산 능력을 가지고 있다. 하지만 스파이킹 뉴럴 네트워크는 이미지 분류와 같은 비교적 쉬운 응용에만 주로 사용되고 있으며 얕은 인공 신경망과 간단한 데이터셋에서만 주로 수행되고 있다. 이러한 제약이 존재하는 가장 큰 요인 중 하나는 스파이크 뉴럴 네트워크에 적합한 학습 알고리즘이 아직 존재하지 않기 때문이다. 스파이크로 정보를 전달하고 연산을 수행하기 때문에 미분이 불가능하다. 따라서 딥 뉴럴 네트워크에서 주로 사용되는 역전파 알고리즘의 사용이 불가능하다. 본 논문에서 딥 스파이킹 뉴럴 네트워크를 이미지 분류보다 더 어려운 회귀 문제 (객체 인식)에 적용해 보고, 딥 뉴럴 네트워크의 성능에 버금가는 객체 인식 모델을 스파이킹 뉴럴 네트워에서 처음으로 제안한다. 더 나아가, 객체 인식 모델의 성능과 지연시간, 에너지 효율성을 향상 시킬 수 있는 여러 방법들을 제안한다. 본 논문은 크게 두 가지 주제로 나누어 설명한다: (a) 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델, (b) 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델의 성능 및 효율성 향상. 제안하는 방법들을 통해 빠르고 정확한 객체 인식 모델을 딥 스파이킹 뉴럴 네트워크에서 수행할 수 있다. 첫 번째 방법은 딥 스파이킹 뉴럴 네트워크에서의 객체 인식 모델이다. 객체 인식 모델은 Spiking-YOLO로 부르고, 저자들이 아는 바에 의하면 PASCAL VOC, MS COCO와 같은 데이터 셋에서 딥 뉴럴 네트워크의 성능에 버금가는 결과를 보여준 첫 번째 스파이킹 뉴럴 네트워크를 기반으로 하는 객체 인식 모델이다. Spiking-YOLO에서는 크게 두 가지 방법을 제안한다. 첫번 째는 채널 별 가중치 정규화이고 두번째는 불균형 한계 전압을 가지는 양음수 뉴런이다. 두 가지 방법을 통해 빠르고 정확한 정보를 딥 스파이킹 뉴럴 네트워크에서 전달 가능하게 한다. 실험 결과, Spiking-YOLO는 PASCAL VOC와 MS COCO 데이터셋에서 딥 뉴럴 네트워크의 객체 인식률의 98%에 뛰어난 성능을 보였다. 또한 Spiking-YOLO가 뉴로모픽 칩에 구현되었음 가정하였을 때, Tiny YOLO보다 약 280의 에너지를 적게 소모하였고 기존의 DNN-to-SNN 전환 방법들 보다 2.3배에서 4배 더 빠르게 수렴하는 것을 확인할 수 있었다. 두 번째 방법은 스파이킹 뉴럴 네트워크에 조금 더 효율적인 연산 능력을 부여하는데 중점을 주고 있다. 비록 스파이킹 뉴럴 네트워크가 희박한 양의 스파이크로 정보를 효율적으로 전달하며 연산 오버헤드와 에너지 소모가 적지만, 두 가지 매우 중요한 문제들이 존재한다: (a) 지연속도: 좋은 성능을 내기 위해 필요한 타임스탭, (b) 시냅틱 연산수: 추론 시 생성된 총 스파이크의 수. 이러한 문제들을 적절히 해결하지 못한다면 스파이킹 뉴럴 네트워크의 큰 장점이라고 할 수 있는 에너지와 전력 효율성이 크게 저하될 수 있다. 이를 해결하기 위해 본 논문에서는 한계 전압 균형 방법론을 새로 제안한다. 제안하는 방법론은 베이시안 최적화 알고리즘을 사용하여 가장 최적의 한계전압 값을 찾는다. 또한 베이시안 최적화 알고리즘을 지연속도나 시냅틱 연산수 등의 스파이킹 뉴럴 네트워크의 특성을 고려할 수 있게 디자인한다. 더 나아가, 두 단계의 한계 전압을 제안하여 높은 에너지 효율을 가지며 더 빠르고 더 정확한 객체 인식 모델을 가능하게 한다. 실험 결과에 따르면 제안하는 방법들을 통해 state-of-the-art 객체 인식률을 달성하였고 기존의 방법들보다 PASCAL VOC에서는 2배, MS COCO에서는 1.85배 빠르게 수렴하는 것을 확인하였다. 또한 시냅틱 연산수도 PASCAL VOC에서는 40.33%, MS COCO에서는 45.31%를 줄일 수 있었다.One of the primary reasons behind the recent success of deep neural networks (DNNs) lies in the development of high-performance parallel computing systems and the availability of enormous amounts of data for training a complex model. Nonetheless, solving such advanced machine learning problems in real world applications requires a more sophisticated model with a vast number of parameters and training data, which leads to substantial amounts of computational overhead and power consumption. Given these circumstances, spiking neural networks (SNNs) have attracted growing interest as the third generation of neural networks due to their event-driven and low-powered nature. SNNs were introduced to mimic how information is encoded and processed in the human brain by employing spiking neurons as computation units. SNNs utilize temporal aspects in information transmission as in biological neural systems, thus providing sparse yet powerful computing ability. SNNs have been successfully applied in several applications, but these applications only include relatively simple tasks such as image classification, and are limited to shallow neural networks and datasets. One of the primary reasons for the limited application scope is the lack of scalable training algorithms attained from non-differential spiking neurons. In this dissertation, we investigate deep SNNs in a much more challenging regression problem (i.e., object detection), and propose a first object detection model in deep SNNs which is able to achieve comparable results to those of DNNs in non-trivial datasets. Furthermore, we introduce novel approaches to improve performance of the object detection model in terms of accuracy, latency and energy efficiency. This dissertation contains mainly two approaches: (a) object detection model in deep SNNs, and (b) improving performance of object detection model in deep SNNs. Consequently, the two approaches enable fast and accurate object detection in deep SNNs. The first approach is an object detection model in deep SNNs. We present a spiked-based object detection model, called Spiking-YOLO. To the best of our knowledge, Spiking-YOLO is the first spiked-based object detection model that is able to achieve comparable results to those of DNNs on a non-trivial dataset, namely PASCAL VOC and MS COCO. In doing so, we introduce two novel methods: a channel-wise weight normalization and a signed neuron with imbalanced threshold, both of which provide fast and accurate information transmission in deep SNNs. Our experiments show that Spiking-YOLO achieves remarkable results that are comparable (up to 98%) to those of Tiny YOLO (DNNs) on PASCAL VOC and MS COCO. Furthermore, Spiking-YOLO on a neuromorphic chip consumes approximately 280 times less energy than Tiny YOLO, and converges 2.3 to 4 times faster than previous DNN-to-SNN conversion methods. The second approach aims to provide a more effective form of computational capabilities in SNNs. Even though, SNNs enable sparse yet efficient information transmission through spike trains, leading to exceptional computational and energy efficiency, the critical challenges in SNNs to date are two-fold: (a) latency: the number of time steps required to achieve competitive results and (b) synaptic operations: the total number of spikes generated during inference. Without addressing these challenges properly, the potential impact of SNNs may be diminished in terms of energy and power efficiency. We present a threshold voltage balancing method for object detection in SNNs, which utilizes Bayesian optimization to find optimal threshold voltages in SNNs. We specifically design Bayesian optimization to consider important characteristics of SNNs, such as latency and number of synaptic operations. Furthermore, we introduce two-phase threshold voltages to provide faster and more accurate object detection, while providing high energy efficiency. According to experimental results, the proposed methods achieve the state-of-the-art object detection accuracy in SNNs, and converge 2x and 1.85x faster than conventional methods on PASCAL VOC and MS COCO, respectively. Moreover, the total number of synaptic operations is reduced by 40.33% and 45.31% on PASCAL VOC and MS COCO, respectively.Abstract i List of Figures ix List of Tables x 1 Introduction 1 2 Background 10 2.1 Object detection 10 2.2 Spiking Neural Networks 16 2.3 DNN-to-SNN conversion 18 2.4 Hyper-parameter optimization 21 3 Object detection model in deep SNNs 25 3.1 Introduction 25 3.2 Channel-wise weight normalization 27 3.2.1 Conventional weight normalization methods 27 3.2.2 Analysis of limitations in layer-wise weight normalization 29 3.2.3 Proposed weight normalization method 30 3.2.4 Analysis of the improved firing rate 38 3.3 Signed neuron with imbalanced threshold 39 3.3.1 Limitation of leaky-ReLU implementation in SNNs 39 3.3.2 The notion of imbalanced threshold 41 3.4 Evaluation 43 3.4.1 Spiking-YOLO detection results 43 3.4.2 Spiking-YOLO energy efficiency 57 4 Improving performance and efficiency of deep SNNs 60 4.1 Introduction 60 4.2 Threshold voltage balancing through Bayesian optimization 62 4.2.1 Motivation 62 4.2.2 Overall process and setup 67 4.2.3 Design of Bayesian optimization for SNNs 69 4.3 Fast and accurate object detection with two-phase threshold voltages 74 4.3.1 Motivation 74 4.3.2 Phase-1 threshold voltages: fast object detection 76 4.3.3 Phase-2 threshold voltages: accurate detection 76 4.4 Evaluation 79 4.4.1 Experimental setup 79 4.4.2 Experimental results 79 5 Conclusion 85 5.1 Dissertation summary 86 5.2 Discussion 88 5.2.1 Overview of the proposed methods and their usages 88 5.3 Challenges in SNNs 90 5.4 Future Work 92 5.4.1 Extension to various applications and DNN models 92 5.4.2 Further improve efficiency of SNNs 93 5.4.3 Optimization of deep SNNs 94 Bibliography 95 Abstract (In Korean) 110Docto

MRL-Seg: Overcoming Imbalance in Medical Image Segmentation With Multi-Step Reinforcement Learning

Medical image segmentation is a critical task for clinical diagnosis and research. However, dealing with highly imbalanced data remains a significant challenge in this domain, where the region of interest (ROI) may exhibit substantial variations across different slices. This presents a significant hurdle to medical image segmentation, as conventional segmentation methods may either overlook the minority class or overly emphasize the majority class, ultimately leading to a decrease in the overall generalization ability of the segmentation results. To overcome this, we propose a novel approach based on multi-step reinforcement learning, which integrates prior knowledge of medical images and pixel-wise segmentation difficulty into the reward function. Our method treats each pixel as an individual agent, utilizing diverse actions to evaluate its relevance for segmentation. To validate the effectiveness of our approach, we conduct experiments on four imbalanced medical datasets, and the results show that our approach surpasses other state-of-the-art methods in highly imbalanced scenarios. These findings hold substantial implications for clinical diagnosis and research

Constrained Twin Variational Auto-Encoder for Intrusion Detection in IoT Systems

Intrusion detection systems (IDSs) play a critical role in protecting billions of IoT devices from malicious attacks. However, the IDSs for IoT devices face inherent challenges of IoT systems, including the heterogeneity of IoT data/devices, the high dimensionality of training data, and the imbalanced data. Moreover, the deployment of IDSs on IoT systems is challenging, and sometimes impossible, due to the limited resources such as memory/storage and computing capability of typical IoT devices. To tackle these challenges, this article proposes a novel deep neural network/architecture called Constrained Twin Variational Auto-Encoder (CTVAE) that can feed classifiers of IDSs with more separable/distinguishable and lower-dimensional representation data. Additionally, in comparison to the state-of-the-art neural networks used in IDSs, CTVAE requires less memory/storage and computing power, hence making it more suitable for IoT IDS systems. Extensive experiments with the 11 most popular IoT botnet datasets show that CTVAE can boost around 1% in terms of accuracy and Fscore in detection attack compared to the state-of-the-art machine learning and representation learning methods, whilst the running time for attack detection is lower than 2E-6 seconds and the model size is lower than 1 MB. We also further investigate various characteristics of CTVAE in the latent space and in the reconstruction representation to demonstrate its efficacy compared with current well-known methods

A Deep Unsupervised Learning Approach for Airspace Complexity Evaluation

Airspace complexity is a critical metric in current Air Traffic Management systems for indicating the security degree of airspace operations. Airspace complexity can be affected by many coupling factors in a complicated and nonlinear way, making it extremely difficult to be evaluated. In recent years, machine learning has been proved as a promising approach and achieved significant results in evaluating airspace complexity. However, existing machine learning based approaches require a large number of airspace operational data labeled by experts. Due to the high cost in labeling the operational data and the dynamical nature of the airspace operating environment, such data are often limited and may not be suitable for the changing airspace situation. In light of these, we propose a novel unsupervised learning approach for airspace complexity evaluation based on a deep neural network trained by unlabeled samples. We introduce a new loss function to better address the characteristics pertaining to airspace complexity data, including dimension coupling, category imbalance, and overlapped boundaries. Due to these characteristics, the generalization ability of existing unsupervised models is adversely impacted. The proposed approach is validated through extensive experiments based on the real-world data of six sectors in Southwestern China airspace. Experimental results show that our deep unsupervised model outperforms the state-of-the-art methods in terms of airspace complexity evaluation accuracy

SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year Anniversary

The Synthetic Minority Oversampling Technique (SMOTE) preprocessing algorithm is considered \de facto" standard in the framework of learning from imbalanced data. This is due to its simplicity in the design of the procedure, as well as its robustness when applied to di erent type of problems. Since its publication in 2002, SMOTE has proven successful in a variety of applications from several di erent domains. SMOTE has also inspired several approaches to counter the issue of class imbalance, and has also signi cantly contributed to new supervised learning paradigms, including multilabel classi cation, incremental learning, semi-supervised learning, multi-instance learning, among others. It is standard benchmark for learning from imbalanced data. It is also featured in a number of di erent software packages | from open source to commercial. In this paper, marking the fteen year anniversary of SMOTE, we re ect on the SMOTE journey, discuss the current state of a airs with SMOTE, its applications, and also identify the next set of challenges to extend SMOTE for Big Data problems.This work have been partially supported by the Spanish Ministry of Science and Technology under projects TIN2014-57251-P, TIN2015-68454-R and TIN2017-89517-P; the Project 887 BigDaP-TOOLS - Ayudas Fundaci on BBVA a Equipos de Investigaci on Cient ca 2016; and the National Science Foundation (NSF) Grant IIS-1447795

Empowering One-vs-One Decomposition with Ensemble Learning for Multi-Class Imbalanced Data

Zhongliang Zhang was supported by the National Science Foundation of China (NSFC Proj. 61273204) and CSC Scholarship Program (CSC NO. 201406080059). Bartosz Krawczyk was supported by the Polish National Science Center under the grant no. UMO-2015/19/B/ST6/01597. Salvador Garcia and Francisco Herrera were partially supported by the Spanish Ministry of Education and Science under Project TIN2014-57251-P and the Andalusian Research Plan P10-TIC-6858, P11-TIC-7765. Alejandro Rosales-Perez was supported by the CONACyT grant 329013.Multi-class imbalance classification problems occur in many real-world applications, which suffer from the quite different distribution of classes. Decomposition strategies are well-known techniques to address the classification problems involving multiple classes. Among them binary approaches using one-vs-one and one-vs-all has gained a significant attention from the research community. They allow to divide multi-class problems into several easier-to-solve two-class sub-problems. In this study we develop an exhaustive empirical analysis to explore the possibility of empowering the one-vs-one scheme for multi-class imbalance classification problems with applying binary ensemble learning approaches. We examine several state-of-the-art ensemble learning methods proposed for addressing the imbalance problems to solve the pairwise tasks derived from the multi-class data set. Then the aggregation strategy is employed to combine the binary ensemble outputs to reconstruct the original multi-class task. We present a detailed experimental study of the proposed approach, supported by the statistical analysis. The results indicate the high effectiveness of ensemble learning with one-vs-one scheme in dealing with the multi-class imbalance classification problems.National Natural Science Foundation of China (NSFC) 61273204CSC Scholarship Program (CSC) 201406080059Polish National Science Center UMO-2015/19/B/ST6/01597Spanish Government TIN2014-57251-PAndalusian Research Plan P10-TIC-6858 P11-TIC-7765Consejo Nacional de Ciencia y Tecnologia (CONACyT) 32901