Ensemble deep learning: A review

Ensemble learning combines several individual models to obtain better generalization performance. Currently, deep learning models with multilayer processing architecture is showing better performance as compared to the shallow or traditional classification models. Deep ensemble learning models combine the advantages of both the deep learning models as well as the ensemble learning such that the final model has better generalization performance. This paper reviews the state-of-art deep ensemble models and hence serves as an extensive summary for the researchers. The ensemble models are broadly categorised into ensemble models like bagging, boosting and stacking, negative correlation based deep ensemble models, explicit/implicit ensembles, homogeneous /heterogeneous ensemble, decision fusion strategies, unsupervised, semi-supervised, reinforcement learning and online/incremental, multilabel based deep ensemble models. Application of deep ensemble models in different domains is also briefly discussed. Finally, we conclude this paper with some future recommendations and research directions

A Comprehensive Survey of Deep Learning in Remote Sensing: Theories, Tools and Challenges for the Community

In recent years, deep learning (DL), a re-branding of neural networks (NNs), has risen to the top in numerous areas, namely computer vision (CV), speech recognition, natural language processing, etc. Whereas remote sensing (RS) possesses a number of unique challenges, primarily related to sensors and applications, inevitably RS draws from many of the same theories as CV; e.g., statistics, fusion, and machine learning, to name a few. This means that the RS community should be aware of, if not at the leading edge of, of advancements like DL. Herein, we provide the most comprehensive survey of state-of-the-art RS DL research. We also review recent new developments in the DL field that can be used in DL for RS. Namely, we focus on theories, tools and challenges for the RS community. Specifically, we focus on unsolved challenges and opportunities as it relates to (i) inadequate data sets, (ii) human-understandable solutions for modelling physical phenomena, (iii) Big Data, (iv) non-traditional heterogeneous data sources, (v) DL architectures and learning algorithms for spectral, spatial and temporal data, (vi) transfer learning, (vii) an improved theoretical understanding of DL systems, (viii) high barriers to entry, and (ix) training and optimizing the DL.Comment: 64 pages, 411 references. To appear in Journal of Applied Remote Sensin

A new Stack Autoencoder: Neighbouring Sample Envelope Embedded Stack Autoencoder Ensemble Model

Stack autoencoder (SAE), as a representative deep network, has unique and excellent performance in feature learning, and has received extensive attention from researchers. However, existing deep SAEs focus on original samples without considering the hierarchical structural information between samples. To address this limitation, this paper proposes a new SAE model-neighbouring envelope embedded stack autoencoder ensemble (NE_ESAE). Firstly, the neighbouring sample envelope learning mechanism (NSELM) is proposed for preprocessing of input of SAE. NSELM constructs sample pairs by combining neighbouring samples. Besides, the NSELM constructs a multilayer sample spaces by multilayer iterative mean clustering, which considers the similar samples and generates layers of envelope samples with hierarchical structural information. Second, an embedded stack autoencoder (ESAE) is proposed and trained in each layer of sample space to consider the original samples during training and in the network structure, thereby better finding the relationship between original feature samples and deep feature samples. Third, feature reduction and base classifiers are conducted on the layers of envelope samples respectively, and output classification results of every layer of samples. Finally, the classification results of the layers of envelope sample space are fused through the ensemble mechanism. In the experimental section, the proposed algorithm is validated with over ten representative public datasets. The results show that our method significantly has better performance than existing traditional feature learning methods and the representative deep autoencoders.Comment: 17 pages,6 figure

Cost-sensitive deep neural network ensemble for class imbalance problem

In data mining, classification is a task to build a model which classifies data into a given set of categories. Most classification algorithms assume the class distribution of data to be roughly balanced. In real-life applications such as direct marketing, fraud detection and churn prediction, class imbalance problem usually occurs. Class imbalance problem is referred to the issue that the number of examples belonging to a class is significantly greater than those of the others. When training a standard classifier with class imbalance data, the classifier is usually biased toward majority class. However, minority class is the class of interest and more significant than the majority class. In the literature, existing methods such as data-level, algorithmic-level and cost-sensitive learning have been proposed to address this problem. The experiments discussed in these studies were usually conducted on relatively small data sets or even on artificial data. The performance of the methods on modern real-life data sets, which are more complicated, is unclear. In this research, we study the background and some of the state-of-the-art approaches which handle class imbalance problem. We also propose two costsensitive methods to address class imbalance problem, namely Cost-Sensitive Deep Neural Network (CSDNN) and Cost-Sensitive Deep Neural Network Ensemble (CSDE). CSDNN is a deep neural network based on Stacked Denoising Autoencoders (SDAE). We propose CSDNN by incorporating cost information of majority and minority class into the cost function of SDAE to make it costsensitive. Another proposed method, CSDE, is an ensemble learning version of CSDNN which is proposed to improve the generalization performance on class imbalance problem. In the first step, a deep neural network based on SDAE is created for layer-wise feature extraction. Next, we perform Bagging’s resampling procedure with undersampling to split training data into a number of bootstrap samples. In the third step, we apply a layer-wise feature extraction method to extract new feature samples from each of the hidden layer(s) of the SDAE. Lastly, the ensemble learning is performed by using each of the new feature samples to train a CSDNN classifier with random cost vector. Experiments are conducted to compare the proposed methods with the existing methods. We examine their performance on real-life data sets in business domains. The results show that the proposed methods obtain promising results in handling class imbalance problem and also outperform all the other compared methods. There are three major contributions to this work. First, we proposed CSDNN method in which misclassification costs are considered in training process. Second, we incorporate random undersampling with layer-wise feature extraction to perform ensemble learning. Third, this is the first work that conducts experiments on class imbalance problem using large real-life data sets in different business domains ranging from direct marketing, churn prediction, credit scoring, fraud detection to fake review detection

Advanced industrial OCR using Autoencoders

Il contenuto di questa tesi di laurea descrive il lavoro svolto durante un tirocinio di sei mesi presso Datalogic ADC. L'obiettivo del lavoro è stato quello di utilizzare uno specifico tipo di rete neurale, chiamata Autoencoder, per scopi legati al riconoscimento o alla convalida di caratteri in un sistema OCR industriale. In primo luogo è stato creato un classificatore di immagini di caratteri basato su Denoising Autoencoder; successivamente, è stato studiato un metodo per utilizzare l'Autoencoder come un classificatore di secondo livello, per meglio distinguere le false attivazioni da quelle corrette in condizioni di incertezza di un classificatore generico. Entrambe le architetture sono state valutate su dataset reali di clienti di Datalogic e i risultati sperimentali ottenuti sono presentati in questa tesi

A Novel Business Process Prediction Model Using a DeepLearning Method

The ability to proactively monitor business pro-cesses is a main competitive differentiator for firms. Processexecution logs generated by process aware informationsystems help to make process specific predictions forenabling a proactive situational awareness. The goal of theproposed approach is to predict the next process event fromthe completed activities of the running process instance,based on the execution log data from previously completedprocess instances. By predicting process events, companiescan initiate timely interventions to address undesired devi-ations from the desired workflow. The paper proposes amulti-stage deep learning approach that formulates the nextevent prediction problem as a classification problem. Fol-lowing a feature pre-processing stage with n-grams andfeature hashing, a deep learning model consisting of anunsupervised pre-training component with stacked autoen-coders and a supervised fine-tuning component is applied.Experiments on a variety of business process log datasetsshow that the multi-stage deep learning approach providespromising results. The study also compared the results toexisting deep recurrent neural networks and conventionalclassification approaches. Furthermore, the paper addressesthe identification of suitable hyperparameters for the pro-posed approach, and the handling of the imbalanced nature ofbusiness process event datasets

Ensemble Learning for fraud detection in Online Payment System: Fraud Detection in Online Payment System

The imbalanced problem in fraud detection systems refers to the unequal distribution of fraud cases and non-fraud cases in the information that is used to train machine learning models. This can make it difficult to accurately detect fraudulent activity. As a general rule, instances of fraud occur much less frequently than instances of other types of occurrences, which results in a dataset which is very unbalanced. This imbalance can present challenges for machine learning algorithms, as they may become biased towards the majority class (that is, non-fraud cases) and fail to accurately detect fraud. In situations like these, machine learning models may have a high accuracy overall, but a low recall for the minority class (i.e., fraud cases), which means that many instances of fraud will be misclassified as instances of something else and will not be found. In this study, Synthetic Minority Sampling Technique (SMOTE) is used for balancing the data set and the following machine learning algorithms such as decision trees, Enhanced logistic regression, Naive Bayes are used to classify the dataset.Majority Voting mechanism is used to ensemble the DT,NB, ELR methods and analyze the performance of the model. The performance of the Ensemble of various Machine Learning algorithms was superior to that of the other algorithms in terms of accuracy (98.62%), F1 score (95.21%), precision (98.02%), and recall (96.75%)

A Stacked Multi-Granularity Convolution Denoising Auto-Encoder

With the development of big data, artificial intelligence has provided many intelligent solutions to urban life. For instance, an image-based intelligent technology, such as image classification of diseases, is widely used in daily life. However, the image in real life is mostly unlabeled, so the performance of many image-based intelligent models shows limitations. Therefore, how to use a large amount of unlabeled image data to build an efficient and high-quality model for better urban life has been an urgent research topic. In this paper, we propose an unsupervised image feature extraction method that is referred to as a stacked multi-granularity convolution denoising auto-encoder (SMGCDAE). The algorithm is based on a convolutional neural network (CNN), yet it introduces a multi-granularity kernel. This approach resolved issues with image unicity by extracting a diverse category of high-level features. In addition, the denoising auto-encoder ensures stability and improves the classification accuracy by extracting more robust features. The algorithm was assessed using three image benchmark datasets and a series of meningitis images, achieving higher average accuracy than other methods. These results suggest that the algorithm is capable of extracting more discriminative high-level features and thus offers superior performance compared with the existing methodologies