DOSSI, MARTINA

MENARDI, GIOVANNA

English

Giovanna Menardi

Archivio istituzionale della ricerca - Università di Padova

Original Citation:AUC-based gradient boosting for imbalanced classificationCentro Editoriale di Ateneo Università di Cassino e del Lazio MeridionalePublisher:Published version:DOI:Terms of use:Open Access(Article begins on next page)This article is made available under terms and conditions applicable to Open Access Guidelines, as described athttp://www.unipd.it/download/file/fid/55401 (Italian only)Availability:This version is available at: 11577/3310317 since: 2020-05-15T14:50:50ZUniversità degli Studi di PadovaPadua Research Archive - Institutional RepositoryAUC-BASED GRADIENT BOOSTING FORIMBALANCED CLASSIFICATIONMartina Dossi1, Giovanna Menardi21 European Central Bank† (e-mail: martina.dossi@ecb.europa.eu)2 Dip. di Sc. Statistiche, Università di Padova (e-mail: menardi@stat.unipd.it)ABSTRACT: Classification problems with imbalanced class distributions are perva-sive in a plurality of real-world applications, such as network intrusion detection, frauddetection and rare disease diagnosis. In this context, most of standard classificationmodels are heavily compromised, as they tend to focus on the majority class, yet theminority class is often the one of greatest importance. To tackle the problem, we com-bine XGBoost, a powerful and recent formulation of the gradient boosting, with a lossfunction specifically derived to optimise the Area Under the ROC curve, an evaluationmetric more robust towards class imbalance.KEYWORDS: AUC, boosting, classification, class imbalance1 IntroductionClass imbalance refers to all supervised classification tasks which suffer ofuneven class distributions. The issue has gained ground with some furtherimplicit assumptions, such that imbalanced data are expected to have rare in-stances belonging to the class of greatest interest and a (relatively) large num-ber of units from the other classes. An imbalanced class distribution mayseverely affect the performance of classification algorithms, by interfering withboth model estimation and accuracy evaluation phases. Disregarding eachmodel own specificities, model estimation is typically driven by the optimi-sation of a global loss function, which favours classification rules ignoring therare units as overwhelmed by the prevalent class. A number of techniques havebeen developed to cope with imbalanced classes: data level approaches attemptto re-balance the class distribution before building learning models, whereasclassifier level approaches aim to adapt existing algorithms to focus on the mi-nority class. The latter group includes cost-sensitive techniques, methods thatreplace the loss function with more meaningful measures and combinations ofclassifiers, that follow the logic of boosting, bagging and random forests.†Disclaimer: this document reflects authors’ views, not necessarily shared by ECB.Under imbalanced scenarios, assessing the performance of a classifier playsa role that is at least as crucial as its estimation. Accuracy, which is the mostcommonly used metric for classification tasks, is not sufficient, as it is gov-erned by the majority class. Other performance metrics which account for theclass distribution are preferred in this context, as the G-mean, the F-measure,and especially the Area Under the ROC Curve (AUC). See Menardi & Torelli(2014) for a more comprehensive discussion about the imbalance problem.Within the logic of the approaches at a classifier level, in this work wederive a differentiable loss function that optimises the AUC to train a gradient-based model within the boosting family, in order to extend the benefits of theAUC as evaluation metric to the phase of model estimation. After presentingthe building blocks relevant for a full comprehension of the proposed method,we discuss our contribution and show some numerical results.2 Gradient boosting optimisation based on the AUCGiven a training set Tn containing n i.i.d. pairs (xi,yi), i = 1, . . . ,n, wherexi ∈ Rd is a vector of attributes and yi ∈ {Y0,Y1} is a response variable whoseclasses are conventionally labeled as negative and positive respectively, a clas-sifier H : X 7→ R is a function that allows to predict the response variable y,based on the observed x. The output H (x) measures the confidence of x be-longing to the positive class, whereas the predicted label ŷ is defined on thebasis of a threshold k ∈ R such that ŷ = Y0 if H (x)< k and ŷ = Y1 otherwise.A non-negative loss function L(y, ŷ), that measures the discrepancy betweenobserved and fitted values, is used either to optimize the classifier during thelearning process and to assess the performance of the model.Even if not specifically developed to tackle the class imbalance problem,the gradient boosting (Friedman, 2001) has showed to achieve competitiveresults in this domain. In broad terms, it exploits the connection between Ad-aBoost, the first applicable approach of boosting, that relies on the idea ofincreasing the weight of the hardest to classify units, and a forward-stagewiseadditive modeling approach. At each iteration of the algorithm, a functionalgradient descent optimisation is applied to a loss function, in the n-dimensionalspace of the fitted values, and it is then approximated by some simple model.The final rule is a linear combination of all the previous estimated functions.A specific formulation of the gradient boosting is XGBoost (Chen & Guestrin,2016), which, at each iteration, approximates the objective loss function by asecond order Taylor’s series expansion, and estimates a classification modelvia its minimisation. This implementation easily supports different loss func-tions, as it is sufficient to provide the algorithm with its first two derivatives.The rationale behind the proposed approach is to integrate into the XG-Boost a loss function independent on the class distribution. In this perspective,the AUC - its ones’ complement, in fact - represents a sensible candidate.Let n+ and n− be the sample size of positive and negative observations re-spectively, and assume that H (x+i ) and H (x−j ) are the fitted scores respectivelyfor the i-th positive and the j-th negative instances. The AUC is equivalent tothe normalized Wilcoxon Mann-Whitney statistic, in the form:AUC =1n+n−n+∑i=1n−∑j=1I0.5(H (x+i )−H (x−j )), (1)where I0.5(t) is 0 if t < 0, 0.5 if t = 0, 1 otherwise. The AUC estimates theprobability that a positive unit receives a higher score than a negative one bymeans of comparisons between instances belonging to different classes. Whilethe global accuracy of a classifier depends on the choice of a classificationthreshold, the AUC evaluates its discriminating ability as the threshold variesover all its range. This allows to cater for the presence of rare units as, byconstruction, it does not place more emphasis on one class over the other.Unfortunately, two issues prevent the expression (1) from being directlyused as a loss function: first and foremost, the function is non differentiable,secondly, its argument is not the single observation but rather refers to pairs ofinstances. To overcome the first limitation, we consider the following differ-entiable approximation (Yan et al., 2003):Us =1n+n−n+∑i=1n−∑j=1S(H (x+i ),H (x−j )), where: (2)S(H (x+i ),H (x−j ))={(−(H (x+i )−H (x−j )− τ))p if H (x+i )−H (x−j )< τ,0 otherwise,(3)for a given τ ∈ (0,1] and p > 1 selected by the user. A pair of observationscontributes to the loss function when the score of a positive unit exceeds theone of a negative unit by τ. The authors suggest to choose τ ∈ [0.1,0.7] andp ∈ {2,3}. The quantity Us is then reformulated to refer to unique instances:Us =1n+n−n∑i=1[I(yi=1)i−1∑i′=1S+i′ + I(yi=−1)i−1∑i′=1S−i′], where: (4)S+i′ = I(yi′=−1)S(H (xi),H (xi′)) and S−i′ = I(yi′=1)S(H (xi′),H (xi)). Once theparameters are defined, the computation of the first two derivatives is straight-forward and the method can be implemented.Empirical results reveal that the proposed approach outperforms many othercompetitive classifiers, especially in scenarios of extreme rarity and nontrivialdata patterns. In the bidimensional setting illustrated in Figure 1, as well as inits generalisation in 5 dimensions, rare units lie in small disjunct sets, over-lapping with the majority class at the margins of each box. The results of theanalysis are outlined in Table 1. As expected, standard models as the logis-tic regression and the classification tree fail in this domain. The algorithmSMOTEBoost (Chawla et al., 2003), specifically developed to address the im-balance, performs even worse than the original AdaBoost. Conversely, themodified XGBoost achieves better results in the majority of the cases, includ-ing the hardest.Figure 1: Simulated data in thebidimensional space. Red dotsrepresent the rare instances.% dim. LogisticReg.Tree(Gini)Ada-BoostSMOTE-BoostGradientboostingModifiedXGBoost0.620.500 0.500 0.772 0.602 0.782 0.790(0.004) (0.000) (0.047) (0.059) (0.043) (0.041)50.500 0.500 0.721 0.563 0.712 0.736(0.012) (0.000) (0.041) (0.059) (0.040) (0.042)120.500 0.501 0.830 0.632 0.838 0.833(0.003) (0.008) (0.034) (0.059) (0.030) (0.030)50.499 0.501 0.786 0.609 0.777 0.790(0.008) (0.014) (0.032) (0.067) (0.030) (0.031)Table 1: Average AUC (and standard deviation) over 300 MonteCarlo samples of size 1000, with dimension 2 and 5, rare class fre-quency of 0.6% and 1%. For boosting algorithms 200 iterations wereconsidered; for the AUC-based loss function τ = 0.7 and p = 2.ReferencesCHAWLA, N. V., et al. 2003. SMOTEBoost: Improving prediction of the mi-nority class in boosting. European conference on principles of data min-ing and knowledge discovery, Springer, 107-119.CHEN, T. & GUESTRIN, C. 2016. XGBoost: A scalable tree boosting system.Proceedings of the 22nd acm sigkdd international conference on knowl-edge discovery and data mining, ACM, 785-794.FRIEDMAN, J. H. 2001. Greedy function approximation: a gradient boostingmachine. Annals of statistics, 1189-1232.MENARDI, G., & TORELLI, N. 2014. Training and assessing classificationrules with imbalanced data. Data Mining and Knowledge Discovery,28(1), 92-122.YAN, L., et al. 2003. Optimizing classifier performance via an approximationto the Wilcoxon-Mann-Whitney statistic. Proceedings of the 20th Inter-national Conference on Machine Learning, 848-855.

AUC-based gradient boosting for imbalanced classification

https://www.research.unipd.it/bitstream/11577/3310317/2/RareClass_Cladag2019.pdf

AUC-based gradient boosting for imbalanced classification

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università di Padova