Eitrich, T.

Lang, B.

Juelich Shared Electronic Resources

John von Neumann Institute for ComputingParallel Cost-Sensitive Support Vector MachineSoftware for ClassificationTatjana Eitrich, Bruno Langpublished inNIC Workshop 2006,From Computational Biophysics to Systems Biology,Jan Meinke, Olav Zimmermann,Sandipan Mohanty, Ulrich H.E. Hansmann (Editors)John von Neumann Institute for Computing, Ju¨lich,NIC Series, Vol. 34, ISBN-10: 3-9810843-0-6,ISBN-13: 978-3-9810843-0-6, pp. 141-144, 2006.c© 2006 by John von Neumann Institute for ComputingPermission to make digital or hard copies of portions of this work forpersonal or classroom use is granted provided that the copies are notmade or distributed for profit or commercial advantage and that copiesbear this notice and the full citation on the first page. To copy otherwiserequires prior specific permission by the publisher mentioned above.http://www.fz-juelich.de/nic-series/volume34Parallel Cost-Sensitive Support VectorMachine Software for ClassificationTatjana Eitrich1 and Bruno Lang21 Central Institute for Applied Mathematics,Research Centre Ju¨lich, 52425 Ju¨lich, GermanyE-mail: t.eitrich@fz-juelich.de2 Applied Computer Science and Scientific Computing Group,Department of Mathematics, University of Wuppertal, GermanyE-mail: lang@math.uni-wuppertal.de1 Support Vector MachinesSupport vector machines (SVMs) are well-known data mining methods for classificationand regression problems1. Their popularity is mainly due to their applicability in variousfields of data mining, such as text mining2, biomedical research3, and many more. Theiraccuracy is excellent and in many cases they outperform other machine learning methodssuch as neural networks. SVMs have their roots in the field of statistical learning whichprovides the reliable generalization theory4. Several properties that make this learningmethod successful are well-known, e.g. the kernel trick5 for nonlinear classification andthe sparse structure of the final classification function. In addition, SVMs have an intuitivegeometrical interpretation, and a global minimum can be located during the SVM trainingphase. In comparison to genetic algorithms or neural networks, less experience is requiredfor using them, which helps researchers to get started with SVM software quite fast.1-1 ?Figure 1. The problem of unbalanced and cost-sensitive classification.2 Cost-Sensitive Support Vector MachineData sets with different class distributions lead to the effect that conventional machinelearning methods are biased towards the larger class6, 7. To overcome this problem and141to obtain sensitive but also accurate classifiers we extended and improved the standardSVM formulation. In addition we use techniques addressing the problem of unbalancedclassification, such as oversampling and threshold moving8. In Fig. 1 an unbalanced toyproblem is shown. Recent results for a CYP P450 drug classification problem are given inref 9.Parallel Support Vector MachineThe main drawback of current SVM models is their high computational complexity forlarge data sets10. This can in fact restrict the applicability of SVMs since the amountof data for classification modeling increases dramatically. Therefore the development ofhighly scalable parallel SVM algorithms is a new important topic of current SVM research.Some algorithms for parallel SVM learning already do exist, but most of them are limited toheuristics for distributed training on reduced data sets11, 12. These are not useful as stand-alone systems for high quality learning on large data. We have implemented a parallelsupport vector machine software well suited for multi-processor shared memory (SMP)clusters that become more and more available. Our algorithm can be used in serial andparallel mode. The parallel implementation provides pure MPI and OpenMP modes aswell as a hybrid mode which combines fine and coarse grained parallelization aspects to awell scalable SVM learning method13. The fine grained inner parallel scheme is shown inFig. 2.create newworking setupdate active kernel matrixfor QP problemgradientupdate and parameterstraining data classi!er!nalcheckconvergenceOpenMPOpenMPParallel SVM Trainingsubproblemsolve new QPESSLSMPFigure 2. Parallel training algorithm based on a decomposition method for SVM training.Summary and Future WorkWe obtained a flexible cost-sensitive parallel SVM software that can be used on high-endmachines with SMP architectures to process the large data sets that arise more and morein bioinformatics and other fields of research. Future work will be on enhanced parametertuning.142AcknowledgmentsThe computations were performed on the IBM p690 cluster JUMP at Research CentreJu¨lich. The authors would like to thank the ZAM team at Ju¨lich for support.References1. N. Cristianini and J. Shawe-Taylor. An introduction to support vector machines andother kernel-based learning methods. Cambridge University Press, Cambridge, UK,2000.2. T. Joachims. Text categorization with support vector machines: learning with manyrelevant features. In Proceedings of the 10th European Conference on MachineLearning (ECML’98), Chemnitz, volume 1398 of Lecture Notes in Computer Science,pages 137–142. Springer, 1998.3. H. Yu, J. Yang, W. Wang, and J. Han. Discovering compact and highly discriminativefeatures or feature combinations of drug activities using support vector machines. In2nd IEEE Computer Society Bioinformatics Conference (CSB 2003), 11-14 August2003, Stanford, CA, USA, pages 220–228. IEEE Computer Society, 2003.4. V. N. Vapnik. Statistical learning theory. John Wiley & Sons, New York, 1998.5. B. Scho¨lkopf. The kernel trick for distances. In NIPS, pages 301–307, 2000.6. C. Elkan. The foundations of cost-sensitive learning. In Proc. of the 17. InternationalJoint Conference on Artificial Intelligence (IJCAI’01), pages 973–978, 2001.7. M. A. Maloof. Learning when data sets are imbalanced and when costs are unequaland unknown. In Worksop on Learning from Imbalanced Data Sets II, ICML, Wash-ington DC, 2003.8. R. Akbani, S. Kwek, and N. Japkowicz. Applying support vector machines to im-balanced datasets. In Proceedings of the 15th European Conference on MachineLearning (ECML’04), Pisa, Italy, 2004, volume 3201 of Lecture Notes in ComputerScience, pages 39–50. Springer, 2004.9. T. Eitrich, A. Kless, C. Druska, J. Grotendorst, and W. Meyer. Classification of highlyunbalanced CYP450 data of drugs using cost sensitive machine learning techniques.Technical Report FZJ-ZAM-IB-2006-09, FZJ, 2006.10. N. Chen, W. Lu, J. Yang, and G. Li. Support vector machine in chemistry. WorldScientific Pub Co Inc, 2004.11. H. P. Graf, E. Cosatto, L. Bottou, I. Dourdanovic, and V. Vapnik. Parallel supportvector machines: the cascade SVM. In Lawrence K. Saul, Yair Weiss, and Le´onBottou, editors, Advances in Neural Information Processing Systems 17, pages 521–528. MIT Press, Cambridge, MA, 2005.12. J.-X. Dong, A. Krzyzak, and C. Y. Suen. A fast parallel optimization for trainingsupport vector machines. In P. Perner and A. Rosenfeld, editors, Proceedings of3rd International Conference on Machine Learning and Data Mining, pages 96–105,2003.13. T. Eitrich, W. Frings, and B. Lang. HyParSVM – a new hybrid parallel software forsupport vector machine learning on SMP clusters. In Proceedings of the Euro-ParConference, Dresden, 2006.143

Parallel Cost-Sensitive Support Vector Machine Software for Classification