Data classification is one of the main techniques of data mining. Different mathematical programming approaches of the data classification were presented in recent years. A technique that uses polyhedral conic functions (PCF) is an effective method for data classification. We present a modified classification algorithm based on PCF functions. Results of numerical experiments on real-world and synthetic data sets demonstrate that the proposed approach is efficient for solving binary data classification problems

Nur Uylaş Satı

Düzce Üniversitesi Bilim ve Teknoloji Dergisi

Directory of Open Access Journals

152   Research Article  A Binary Classification Algorithm Based on Polyhedral Conic Functions Nur Uylaş SATI* Department of Mathematics, Faculty of Science, Ege University, İzmir, TURKEY * Corresponding author’s e-mail address: nuruylas@gmail.com     ABSTRACT Data classification is one of the main techniques of data mining. Different mathematical programming approaches of the data classification were presented in recent years. A technique that uses polyhedral conic functions (PCF) is an effective method for data classification. We present a modified classification algorithm based on PCF functions. Results of numerical experiments on real-world and synthetic data sets demonstrate that the proposed approach is efficient for solving binary data classification problems. Keywords: Mathematical Programming, Polyhedral Conic Functions, Classification, Clustering.   Çokyüzlü Konik Fonksiyonlar Temelli Bir İkili Sınıflandırma Algoritması ÖZET Veri sınıflandırma, veri madenciliğinin önemli tekniklerinden birisidir. Son yıllarda veri sınıflandırması için farklı matematiksel programlama yaklaşımları sunulmuştur. Çokyüzlü konik fonksiyonları kullanan bir teknik veri sınıflandırması için efektif bir yöntem olmuştur. Bu çalışmada çokyüzlü konik fonksiyonları temel alan geliştirilmiş bir sınıflandırma algoritması sunulmuştur. Gerçek hayat ve sentetik veri kümeleri üzerinde yapılan sayısal deney sonuçları göstermektedir ki sunulan yaklaşım ikili veri sınıflandırma problemlerinin çözümünde etkili olmuştur. Anahtar Kelimeler: Matematiksel Programlama, Çokyüzlü Konik Fonksiyonlar, Sınıflandırma, Kümeleme.    Received: 23/12/2014, Revised: 07/01/2015, Accepted: 07/01/2015  Düzce University  Journal of Science & Technology Düzce University Journal of Science and Technology, 3 (2015) 152-161  153 I. INTRODUCTION  HE supervised data classification uses data whose classes are known. These data sets are called training sets. The aim of supervised data classification is to define rules on this training set. By using these rules making efficient data classification is expected. The efficiency of the found rule is examined on the test set. Supervised data classification applications can be met up in every area that involves data mining such as business, medicine, engineering etc. Binary classification problem consists of finding an appropriate surface in IRn separating two discrete point sets and it is particularly based on mathematical programming. Several mathematical programming techniques for binary classification problems were used in [2,3,8,7,9,12]. Some of these techniques are mentioned in the next section. In this paper an algorithm based on the PCF functions is presented. Firstly we aim to find efficient vertices of cones thus we can develop the algorithm performance and secondly to prevent overfitting because of the difference between training and testing accuricies. In accordance with this purpose the algorithm based on PCF is reformulated, new constraints and methods are added. Numerical experiments have been carried out. In conclusion the results show that the proposed method improved the approach based on PCF for binary data classification.  The rest of the paper is organized as follows: A brief description of binary classification is given in section 2. In section 3, polyhedral conic functions and basic properties of them are given. Besides the PCF algorithm is expressed. In section 4 a new formulation for binary data clasification model is given and an algorithm based on this model is presented. The results of numerical experiments are given in section 5 and finally section 6 concludes the paper.  II. METHOD  A. BINARY CLASSIFICATION  Binary classification problem can be stated as follows: A random couple   , where    is called the feature vector is given and   is called the label. The goal is to learn a classifier, i.e., a mapping   that separates the vectors with reference to labels [7].  Verbally, it can be defined as a problem of obtaining a criterion for distinguishing between the elements of two disjoint sets of patterns [13]. Many algorithms have been proposed and studied to solve this problem [2, 3, 5, 6, 7, 8, 9, 12, 13, 14, 15] in the last decades. Most frequently used ones are based on linear, polyhedral and max-min separation.  In paper [14] pattern separation problem which is a binary separation problem is solved as a convex programming problem. In [13] the same idea is used and to achieve separation a plane or a non linear surface, such that one set of patterns lies on one side of the plane and the other set of patterns on the other side, is constructed. In [8] Bennett and Mangasarian presented a method to find such a hyperplane. This method is based on linear separability. In linear separability the convex hulls of the two sets do not intersect. If the intersection is not empty a hyperplane can be constructed by letting some misclassification or nonlinear separating surfaces can be looked for. T154 The concept of polyhedral separability was introduced in [2]. Astorino and Gaudiso used h hyperplanes, that configures a convex polyhedron, for binary classification. They introduced an error function which is piecewise linear but not convex nor concave. In [3] Bagirov described max-min separability that is a generalization of h-polyhedral separability. It solves the problem by finite number of hyperplanes that constructs piecewise linear function. And it is proved that if the intersection of two sets is empty they can be strictly separated by a max-min of linear functions . In this paper PCF, a method that is defined in [9], is used for binary classification. In the next section this method will be analyzed. B. POLYHEDRAL CONIC FUNCTIONS(PCF) Polyhedral conic functions (PCFs) have recently been proposed to separate two disjoint point sets in nIR  [9] . In [9] Definition 1 and Lemma 1 quoted below are given and proofed.  Definition 1: A function : ng IR IR   is called polyhedral conic if its graph is a cone and all its level sets,  : ( ) ,nS x IR g x IR         are polyhedrons.  Given '1 1, , , , ...nn nw a IR IR w x w x w x       is a scalar productof and w x , 11 ... nx x x    is a l1 norm of the vector nx IR , a polyhedral conic function  ( , , , ) :nw ag IR IR    defined as                     ( , , , ) 1: ( )nw ag IR IR w x a x a                                                         (1) Lemma 1: A graph of the function ( , , , )w ag   defined in (1) is a polyhedral cone with a vertex at ( , ) na IR IR   . This cone is called a polyhedral conic set and a its center. Let A and B be given sets containing m and p n-dimensional vectors, respectively:  , ,  i nA a R i I    ,j nB b R j J   where    1,..., , 1,...,I m J p  . An algorithm generating a polyhedral conic separating function, called a PCF algorithm, proceeds as follows [9]: Algorithm 1. PCF Algorithm for binary data classification. Step 0.( Initialization step) l=1, ,l lI I A A   and go to Step 1.  Step 1. Let la be an arbitrary point of  lA . Solve subproblem lP.                                                '( )  min( )lIlly ePI                                                                                   (2) 155                                            '1( ) 1 ,     ,i l i l i lw a a a a y i I                                             (3)                                           '1( ) 1 0,     ,j l j lw b a b a j J                                               (4)                                              1( ,..., ) , , , 1m nmy y y R w R R                                                    (5) Let  , , ,l l l lw y  be a solution of ( )lP  and let                                                        ( , , , )( ) ( )l l l ll w ag x g x                                                                     (6) and go to Step 2. Step 2.  Let    1 1 1: ( ) 1 0 , : , 1i il l l l l lI i I g a A a A i I l l           and if  lA                                           go to Step 1. Step 3. Define the function  g(x) (separating the sets A and B) as                                                                     ( ) min ( )llg x g x                                                               (7) and stop. In this algorithm the number of iterations causes efficiency decreasing and it is strongly depends on the place of the vertex of polyhedral cones, ( , )l l na IR IR   . To solve this problem in [9] a modified PCF algorithm is suggested. It finds the al point in a more efficient way when the set A under consideration is not too large. In the modified PCF algorithm at each iteration l, the problem (Pl) is solved for each l lia A and the numbers of elements  li from Al separated from B are found. Then al is defined as 0lla a where  0l   max :i ll i I  [9]. C. A BINARY CLASSIFICATION ALGORITHM The modified PCF algorithm defined in the previous section is more efficient than the old one but it has a hole when one of the training set A under consideration is too large. It causes more iterations, at each iteration l, Pl is solved for each ia A .   We solve this problem by using clustering methods. These useful methods for analysis of patterns in data, are offered by data mining, in particular machine learning algorithms. Cluster analysis algorithms form groups of objects that share common properties[11]. Several algorithms have been studied for clustering method [1]. In this paper we use one of respected, k-means algorithm.  Given a set of observations 1( ,..., )mx x where each observation is a d-dimensional real vector, k-means clustering aims to partition the m observations into k sets (k ≤ m)  1,..., kS S S  so as to minimize the within-cluster sum of squares (WCSS) [4]: 156 2arg minj ij iS x Sx   where i  is the mean of points in iS .  The k-means algorithm proceeds as follows [4]: Algorithm 2.  k-means algorithm. Step 1. Choose a seed solution consisting of k centers (not necessarily belonging to A); Step 2. Allocate data points ia A  to its closest center and obtain k-partition of A; Step 3. Recompute centers for this new partition and go to Step 2 until no more data points change cluster.  In modified PCF algorithm, mentioned in the previous section, the al center points are chosen according to the number of elements Al separated from B. In constraint (3), obtaining maximal number of elements is related to the closeness of Al points to al. Therefore instead of solving  Pl  for each l lia A to find an optimal center in every iteration in modified PCF, we initially use k-means clustering method and obtain  k  numbers of  ak  optimal centers that are the closest ones to the corresponding Ak points with reference to the k-means method. Then solve k numbers of Pl subproblems for each ak in PCF algorithm.   Besides, in this paper we change the constraint (4) of  Pl subproblem for decreasing large differences between accuricies on training and test sets, and we aim to prevent over-fitting the classification problem and get a good generalization. We apply relaxation to this constraint by allowing some misclassification as follows;  1( ) 1 ,   j jk k jw b a b a z j J          where 0jz   is a slack variable that measures how much a B point fails to be outside of the polyhedron corresponding to the sublevel set  : ( ) 1lx g x  . If 0jz   , there is no misclassification and if 1jz  , bj point is on the polyhedron.  Thus, we construct Pl  subproblem as follows;  1 11 1minpmi ji iy C zm p     1( ) 1 ,   i ik k iw a a a a y i I         1( ) 1 ,   j l j l jw b a b a z j J          , 0, 1, , , 1 ni jy z C w R R       where C   1 is the fixed penalty parameter, used for the misclassification of jb B  points, that is predefined.  157 Then, a binary classification algorithm based on clustering and PCF functions defined as follows: Let A and B two given sets in            : , :i n j nA a IR i I B b IR j J       where    1,.., ,    1,..,I m J p  . Algorithm 3. PCF algorithm with clustering for binary classification  Step 0.( Initialization step) Apply clustering algorithm on  set of A. Let s be the number of clusters and k=1 . Ik=I. Step 1. Let       be the center of k th cluster . Solve subproblem     1 11 1( )     minpmk i ji iP y C zm p     1( ) 1 ,   i ik k i kw a a a a y i I         1( ) 1 ,   j jk k jw b a b a z j J          , 0, 1, , , 1 ni jy z C w R R       Let , , ,k k k kw y   be a solution of ( )kP . Let  ( , , , )( ) ( )k k k kk w ag x g x   Step 2. If k s , let k=k+1 ,   1 1: 0ik k kI i I g a     and go to Step 1. Step 3.  Define the function g(x) (separating the sets A and B) as  ( ) min ( )kkg x g x  and stop.  III. RESULTS & DISCUSSION We present the efficiency of the presented algorithm by carrying out numerical experiments with a number of real world and synthetic datasets. MATLAB is used for applications. We compare proposed algorithm and PCF algorithm due to their accurices and time. The results are shown in tables.   ka kPnIR158 Accuracy is defined as the ratio between the number of well classified points of both A and B as follows:  wc: number of  well classified points of A and B te: number of training set elements  Accuracy=100 wcte  In the proposed algorithm, k-means method was used for clustering. k(1-20) was defined as to get the best accuracy. C penalty number was defined as 10 to allow less misclassification to jb B  points than ia A  points. The accuricies are not 100 as in modified PCF [9] because of allowing misclassification to jb B  points, and stopping the algorithm  at the k th (number of clusters) iteration. Table 1 shows the number of instances and attributes of the datasets used. The results shown in Table 2 indicates that the proposed algorithm is more efficient with regard to time. Accuracy values are not 100% as PCF because of allowing some misclassification and stopping the algorithm in k (defined in clustering method) iterations. We terminate the algorithm if time exceeds 1800 sec. and show it with          (-). Besides on the same datasets for testing the validities of new algorithm and PCF algorithm, ten-fold cross validation tests are applied. Ten-fold cross validation is explained as followsin [10]; the dataset D is randomly split into 10 mutually exclusive subsets (the folds) D1,D2,..,D10 of approximately equal size. The inducer is trained and tested 10 times; each time  1, 2,...,10t , it is trained on D\Dt and tested on  Dt. The cross validation estimate of accuracy is the overall number of correct classifications, divided by the number of instances in the data set [10].  Table 1. The brief description of real world data sets  Data sets Number of instances Number of attributes Blood Transfusion Ionosphere Fertility WBCD Heart Connectionist Bench 748 351 100 683 297 208 5 34 10 9 13 60      159 Table 2. Results of real-world data sets obtained using Algorithm 3 and PCF ALGORITHM 3  PCF Data sets Accuracy % Time Sec.  Accuracy % Time Sec. Blood Transfusion 60.45 338 sec.  - - Ionosphere 98.86 42 sec.  100 756 sec. Fertility 90 22 sec.  100 35 sec. WBCD 97.28 35 sec.  100 1763 sec. Heart 85.90 22 sec.  100 613 sec. Connectionist Bench 83.80 84 sec.  100 287 sec.    Table 3. Tenfold cross-validation results of real-world data sets obtained using Algorithm 3 ALGORITHM 3  PCF Data sets Training Accuracy % Testing Accuracy %  Training Accuracy % Testing Accuracy % Blood Transfusion 80.56 78.55  - - Ionosphere 98.10 94.28  100 95.76 Fertility 90.22 80.23  100 90.45 WBCD 98.21 98.55  100 100 Heart 88.88 91.66  100 86.95 Connectionist Bench 100 96.79  100 80.38    Table 4. The brief description of synthetic data sets Data sets Number of instances Number of attributes Dataset 1 Dataset 2 Dataset 3 Dataset 4 20 50 100 300 6 6 6 6    Table 5. Accuracy results of synthetic datasets   Modified PCF [17] Algorithm 3 Data sets accuracy Time(sec) accuracy Time(sec) Dataset 1 100 0.38 100 0.16 Dataset 2 100 1.70 100 0.11 Dataset 3 100 6.72 100 2.76 Dataset 4 100 211.09 100 0.92    160 Table 6. Tenfold cross-validation results of synthetic data sets obtained using algorithm 3  Modified PCF in[17]  Algorithm 3 Data sets Training Accuracy % Testing Accuracy %  Training Accuracy % Testing Accuracy % Dataset 1 100 90  100 95 Dataset 2 100 96 100 100 Dataset 3 100 100  100 100 Dataset 4 100 100  100 100  As can be seen from Table 3 the large differences between accuracies on training and test sets can be reduced by letting some amount (defined by the C fixed penalty parameter) of misclassification to B points.  Also in Table 5 and Table 6 same comparisons are applied to 4 synthetic datasets, whose number of attributes and instances are shown in Table 4.    IV. CONCLUSION In this paper, an algorithm is developed for solving the binary classification problems. Algorithm is based on clustering and PCF functions for separating the given finite point sets in n-dimensional space. The proposed algorithm retrieves overfitting the classification problem by letting misclassifications and allows saving time while solving problems with large datasets by the help of clustering method.    V. REFERENCES  [1]   M.R. Anderberg,  Cluster Analysis for Applications, Academic Press, New York  (1973).  [2] A. Astorino, M. Gaudioso, Journal of Optimization Theory and Applications 112(2) (2002) 265.  [3]  A.M. Bagirov, Optimization Methods and Software 20(2-3) (2005) 277.   [4]  A.M. Bagirov, K. Mardaneh, WISB '06 Proceedings of the 2006 workshop on Intelligent systems for bioinformatics (73)  (2006) 23.   [5]  A.M. Bagirov, J. Ugon  (2005) DOI: 10.1007/0-387-26771-9_6.   [6]  A.M. Bagirov, J. Ugon, D. Webb, B. Karasözen, Pattern Analysis and Applications 14(2) (2011) 165.   [7]  A.M. Bagirov, J. Ugon, D. Webb, G. Öztürk, R. Kasımbeyli (2011) DOI: 10.1007/s11750-011-0241-5.  [8]  K.P. Bennett, O.L. Mangasarian,  Optimization methods and software 1(1)  (1992) 23.   [9]  R.N. Gasimov, G. Öztürk  Optimization Methods and Software  21(4) (2006) 527.   [10]  R. Kohavi, A study of cross-validation and bootstrap for accuracy estimation and model selection, International Joint Conference on Artificial Intelligence, San Francisco (1995) 1137.   [11]  A. Kusiak (2001) DOI:10.1117/12.417237.  161  [12]  J.M. Liittschwager, C. Wang,  Management Science 24(14)  (1978) 1515.  [13]  O.L. Mangasarian, Operations Research 13(3)  (1965) 444.   [14]  J.B. Rosen,  Journal of Mathematical Analysis and Applications 10(1) (1965) 123.  [15]  V. Vapnik, The nature of statistical learning theory,  Springer Verlang,  New York (1995).  

A Binary Classification Algorithm Based on Polyhedral Conic Functions

https://dergipark.org.tr/tr/download/article-file/56335

A Binary Classification Algorithm Based on Polyhedral Conic Functions

Abstract

Similar works

Full text

Available Versions

Directory of Open Access Journals