制度:新 ; 報告番号:甲3136号 ; 学位の種類:博士(工学) ; 授与年月日:2010/7/12 ; 早大学位記番号:新541

Li Boyang

English

DSpace at Waseda University

 早稲田大学大学院情報生産システム研究科     博 士 論 文 概 要         論 文 題 目  Study on Multi-SVM Systems and Their Applications to Pattern Recognition        申  請  者   LI, Boyang   情報生産システム工学専攻 ニューロコンピューティング研究      2010 年 5 月  2 Kerne l -based  t echn iques  r ep re sen t  a  ma jo r  deve lopmen t  i n  mach ine  l ea rn ing  a lgo r i thms .  As  one  o f  the  mos t  wide ly  used  ke rne l -based  t echn ique ,  Suppor t  Vec tor  Machine  (SVM)  has  a t t r ac t ed  a  l o t  o f  i n t e r es t  f rom resea rcher s  i n  r ecen t  yea r s .  SVMs a re  a  g roup  o f  supe rv i sed  l ea rn ing  me thods  tha t  can  be  app l i ed  to  pa t t e rn  r ecogni t i ons  ( c l a s s i f i ca t i on  o r  r eg re ss ion ) .  Numerous  books  a re  ava i l ab le  fo r  an  in -dep th  ove rv iew o f  t he  t heore t i ca l  bas i s  o f  t hese  a lgor i thms .   In  compar i son  wi th  o the r  me thods  in  mos t  cu r ren t  l i t e r a tu res ,  t he  SVM-based  me thods  show advan tages  in  t e r ms  o f  gene ra l i za t i on  pe r fo rmance  and  the  f i t nes s  accuracy .  However ,  t he re  s t i l l  ex i s t  some  cha l l enges  when  app ly ing  SVMs to  the  r ea l -wor ld  pa t t e rn  r ecogn i t ion  p rob lems :  ?  Large  t ra in ing  data  Because  SVM i s  one  o f  t he  ke rne l -based me thods  and  fo rmula t ed  a s  quadra t i c  p rogramming  (QP)  p rob lems ,  i t s  t r a in ing  t ime  and  space  complex i t i e s  have  exponen t i a l  r e l a t ionsh ips  wi th  the  s i ze  o f  t r a in ing  da ta .  Hence ,  a  ma jo r  s tumbl ing  b lock  in  SVM i s  t he  h igh  t r a in ing  complex i ty  fo r  l a rge  t r a in ing  da ta .  ?  High  d imens iona l  input  Depend ing  on  the  acqu is i t i on  r e so lu t ion ,  many  rea l -wor ld  da tabases  (mic roa r r ay ,  t ex t  c l a s s i f i ca t i on ,  e t c . )  cons i s t  o f  a  number  o f  measuremen t s .  Whi l e  the  h igh  d i mens ion  o f  the  da tabase  po ten t i a l ly  makes  c l a s s i f i e r  a  un ique  and  power fu l  t echn ique  fo r  a  ce r t a in  app l i ca t ion ,  on  the  o the r  hand  i t  compl i ca t e s  t he  computa t ion  and  the  des ign  o f  an  appropr i a t e  me thod .  ?  Noise  and  in terac t ion  Since  the  acqu i s i t i ons  used  in  th i s  work  ( t r a in ing  da ta  and  t e s t  da t a )  a r e  ob ta ined  f rom the  rea l -wor ld ,  t he  da t abases  a re  usua l ly  a f fec t ed  by  in t e r ac t ion  and  no i ses .  In  mos t  ac tua l  des igns  o f  c l a s s i f i e r s ,  no i se s  and  in t e rac t ion  make  the  boundar i e s  be tween  c l a s ses  no t  c l ea r  and  some  samp les  mi sc la s s i f i ed .  ?  Imbalance  o f  data  In  many  ac tua l  pa t t e rn  r ecogni t ion  app l i ca t ions ,  we  have  to  f ace  a  p rob lem o f  imba lance  o f  da t a ,  i . e .  some  c l a s ses  have  much  more  in s t ances  than  o the r s .  Th i s  phenomenon  wide ly  ex i s t s  i n  the  r ea l -wor ld  and  i t  i s  t he  ma in  r eason  fo r  caus ing  the  excur s ion  o f  sepa ra t ion  boundar i e s  in  SVMs.    In  o rde r  to  hand le  the  a fo remen t ioned  cha l l enges  inhe ren t  i n  t he  c l a ss i f i ca t ion  and  r eg res s ion  p rob lems ,  t h i s  work  ma in ly  p roposes  th ree  improved  approache s  in  th ree  impor t an t  s t ages  (p reprocess ing / fea tu re  s e l ec t ion / recogn i t i on )  o f  t he  typ ica l  pa t t e rn  r ecogn i t ion  sys t em re spec t ive ly :  Fas t  SVM Tra in ing  Us ing  Edge  De tec t ion ,  Feature  Se lec t ion  Us ing  Corre la t ion-based  SVM Fi l t e r  and  3 Class i f i ca t ion  Us ing  Fuzzy  Dec i s ion-making  SVM .    I n  o rde r  to  ob ta in  be t t e r  pe r fo rmances ,  t h i s  work  a l so  p roposes  two  mul t ip l e  SVM pa t t e rn  r ecogn i t ion  sys t ems :  Mul t ip l e  Suppor t  Vec tor  C lass i f i e r  Sys t em  and  Mul t ip l e  Suppor t  Vec tor  Regress ion  Ne twork .   The  in t roduc t ions  and  de ta i l s  o f  t hese  p roposed  approaches  a re  desc r ibed  in  t h i s  t hes i s .  The  thes i s  cons i s t s  o f  seven  chap te r s .   Chapter  1  br i e f ly  in t roduces  pa t te rn  r ecogn i t ion  sys t e ms ,  r ea l -wor ld  p rob lems ,  ou r  mot iva t ion ,  goa l s  o f  t h i s  work  and  ou t l i ne  o f  the  thes i s .   Chapter  2  p roposes  a  f a s t  SVM t ra in ing  me thod  based  on  the  edge  de tec t ion  t echn ique  fo r  so lv ing  the  cha l l enge  o f  l a rge  t r a in ing  da t a  i n  t he  p reprocess ing  s t age .  The  p roposed  approach  has  two  ma in  advan tages :  ( i )  s ca l ing  down the  t r a in ing  da ta  obv ious ly  ( i i )  r e se rv ing  Suppor t  Vec to r s  (SVs)  max i ma l ly .  F i r s t ,  t he  edge  de tec t ion  t echn i que  i s  i n t roduced  to  ex t r ac t  s amp les  nea r  the  sepa ra t ion  boundary .  These  samples  a re  l i ke ly  to  be  SVs  in  SVM.  Second ,  to  avo id  the  ove r f i t t i ng  and  p rese rve  the  s t ruc tu re  p rope r t i e s ,  we  a l so  run  c lus t e r ing  in  the  whole  t r a in ing  da ta .  The  r econs t ruc ted  t r a in ing  da ta  cons i s t s  o f  s amples  s e l ec t ed  by  edge  de tec to r  and  cen t ro ids  o f  c lus t e r s .  I t  i s  muc h  sma l l e r  t han  the  o r ig ina l  t r a in ing  da ta ,  and  makes  the  SVM t ra in ing  p rocess  much  fa s t e r ,  bu t  w i thou t  deg rad ing  the  c l a s s i f i ca t i on  accurac ie s .    Chapter  3  r epor t s  a  f ea tu re  s e l ec t i on  based  on  cor re l a t i on-based  fea tu re  c lus t e r ing  and  SVM-based  f ea tu re  r ank ing  fo r  so lv ing  the  cha l l enge  o f  h igh  d i mens iona l  i npu t .  Th i s  me thod  focuses  on  f ind ing  the  good  fea tu re  se t  cons i s t ing  o f  f ea tu res  tha t  a r e  h igh ly  co r re l a t ed  wi th  the  ou tpu t ,  ye t  uncor re l a t ed  wi th  each  o the r .  F i r s t ,  t he  f ea tu re  sens i t i v i ty  in  SVM i s  de f ined  to  eva lua te  the  in f luence  quan t i ty  o f  a  f ea tu re  on  the  ou tpu t .  Second ,  f ea tu res  a re  r anked  based  on  the i r  s ens i t iv i t i e s ,  and  the  ones  wi th  low sens i t iv i t i e s  a r e  e l i mina ted .  Th i rd ,  a  cor re l a t ion-based  c lus t e r ing  i s  imp lemented  to  d iv ide  the  f ea tu res  i n to  some  c lus t e r s .  Then ,  we  se lec t  t he  f ea tu res  t ha t  a r e  r anked  f i r s t  i n  the  c lus t e r s  a s  t he  se l ec t ed  s ign i f i can t  f ea tu res .  In  t he  cor re l a t ion -based  c lus t e r ing ,  co r re l a t ion  coe f f i c i en t  and  in fo rma t ion  ga in  func t ion  a re  used  a s  the  c r i t e r ions  fo r  eva lua t ing  the  f ea tu re  co r re l a t ion .   Chapter  4  exp lores  a  c l a ss i f i ca t ion  mode l  u s ing  a  fuzzy  dec i s ion -mak ing  sepa ra t ion  boundary  fo r  so lv ing  the  cha l l enges  o f  no i ses ,  i n t e rac t ion  and  4 imba lance  o f  da t a .  Cons ide r ing  the  vagueness  be tween  the  r ea l -wor ld  da ta  se t s  caused  by  no i ses  and  in t e r ac t i ons ,  a  fuzzy  dec i s ion-mak ing  func t ion  i s  f i r s t  des igned  to  t ake  the  p l ace  o f  t he  c l a s s i ca l  s ign  func t ion  in  the  p red ic t ion  pa r t  o f  SVM.  Th i s  f l ex ib le  des ign  o f  dec i s ion - mak ing  sepa ra t ion  boundary  i s  more  s imi l a r  t o  t he  r ea l -wor ld  env i ronmen t  t han  conven t iona l  SVMs.  Then  an  o f f se t  pa ra me te r  i s  i n t roduced  in to  t he  fuzzy  dec i s ion-making  func t ion  to  mod i fy  the  excur s ion  o f  sepa ra t ion  boundary  caused  by  the  imba lance  o f  da t a .  The  o f f se t  i s  ca l cu l a t ed  a s  t he  Weigh ted  Harmon ic  Mean  (WHM) o f  t he  dec i s ion  va lues  o f  SVs .  Th i s  me thod  can  r educe  the  in f luence  o f  no i ses  on  the  ca l cu la t ion  o f  o f f se t .   Chapter  5  p roposes  a  mul t ip l e  SVM c la s s i f i e r  sys t em.  I t  has  been  shown by  severa l  r e sea rcher s  t ha t  mul t ip l e  pa t t e rn  r ecogn i t ion  sys t ems  can  r e su l t  i n  e f fec t i ve  so lu t ions  to  d i f f i cu l t  r ea l -wor ld  t a sks .  D i f fe r ing  f rom c la s s i ca l  approaches ,  we  fo r mula te  the  t a sk  o f  c l a s s i f i e r  mode l ing  a s  the  p i ecewise  approx ima t ion  o f  the  sepa ra t ion  boundary .  The  p roposed  mul t ip l e  SVM  c lass i f i e r  sys t e m i s  cons t ruc ted  based  on  p i ecewise  pa r t i t i on  and  in t e rpo la t ion .  F i r s t ,  edge  de tec t ion  me thod  i s  u sed  to  iden t i fy  the  sepa ra t ion  boundary .  Second ,  the  sepa ra t ion  boundary  i s  pa r t i t i oned  in to  some  subse t s  based  on  the  rough  se t  t heo ry .  Based  on  the  pa r t i t i on  o f  sepa ra t ion  boundary ,  t r a in ing  da ta  se t s  a re  cons t ruc ted  to  desc r ibe  the  sub t asks  fo r  i den t i fy ing  segmen t s  o f  t he  sepa ra t ion  boundary .  Then ,  l oca l  SVMs a re  subsequen t ly  t r a ined  to  so lve  the  r e spec t ive  sub tasks .  F ina l ly ,  t he  dec i s ions  o f  t he  loca l  SVMs a re  appropr i a t e l y  combined  based  on  a  p robab i l i s t i c  i n t e rp re t a t ion  to  ob ta in  the  f ina l  c l a s s i f i ca t ion  dec i s ion .    Chapter  6  p roposes  a  mul t ip l e  SVM sys t ems  named  Modu la r  SVR Ne twork  fo r  t ime  se r i e s  p red ic t ion .  I t  i s  a l so  cons t ruc ted  based  on  the  p r inc ip l e  o f  “d iv ide -and-conquer” .  The  p roposed  modu la r  SVR ne twork  i s  cons t ruc ted  based  on  the  p r i ce  domain  pa r t i t i on .  In  the  t r ans fo r ma t ion  l aye r ,  t he  p r i ce  domain  o f  t i me  se r i e s  i s  pa r t i t i oned  in to  seve ra l  sub  p r i ce  r eg ions .  The  p rope r t i e s  o f  t hese  sub  p r i ce  r eg ions  a re  desc r ibed  by  r e spec t ive  SVRs .  The  SVRs  a re  used  a s  “ loca l  expe r t s” .  In  the  p red ic t ion  l ayer ,  t he  ou tpu t s  o f  t hese  SVRs  a re  combined  by  a  combina to r .  Because  the  pa r t i t i on  o f  t r a in ing  da ta  i s  imp lemen ted  in  t he  ou tpu t  space  (p r ice  doma in )  wh ich  has  a  much  sma l l e r  d i mens iona l i t y  t han  the  inpu t  space ,  t he  mode l  i s  eas i e r  t o  be  cons t ruc ted .   Chapter  7  conc ludes  th i s  work ,  summar izes  the  thes i s  and  g ives  some  po ten t i a l  p ro j ec t s  fo r  our  fu r the r  r e sea rch .  

Study on multi-SVM systems and their applications to pattern recognition

制度:新 ; 報告番号:甲3136号 ; 学位の種類:博士(工学) ; 授与年月日:2010/7/12 ; 早大学位記番号:新5414textthesi

Li, Boyang

32127

Waseda University Repository

https://waseda.repo.nii.ac.jp/?action=repository_action_common_download&item_id=19397&item_no=1&attribute_id=20&file_no=3

Study on multi-SVM systems and their applications to pattern recognition

Abstract

Similar works

Full text

Available Versions

DSpace at Waseda University

Waseda University Repository