Presentation of video classification project, including the TRECVID MED2010 system

Ellis, Daniel P. W.

Chang, Shih-Fu

Jiang, Yu-Gang

Zeng, Xiaohong

Ye, Guangnan

Cotton, Courtenay Valentine

English

Columbia University Academic Commons

Joint	  Audio-­‐Visual	  Signatures	  	  for	  Web	  Video	  Analysis Dan Ellis, Shih-Fu Chang Yu-Gang Jiang, Xiaohong Zeng,  Guangnan Ye, Courtenay Cotton  Department of EE, Columbia University, NY NGA	  NARP	  Symposium,	  Aug	  31,	  2011	  •  Original	  unedited	  videos	  made	  from	  consumers	  	    Interes4ng	  and	  very	  diverse	  contents	    Very	  weakly	  indexed:	  	  3	  tags	  per	  consumer	  video	  vs.	  9	  tags	  avg	    Original	  audio	  tracks	  -­‐	  good	  for	  audio-­‐visual	  joint	  analysis	    Challenge:	  Content-­‐based	  retrieval	    Find	  items	  similar	  to	  example(s)	  What	  are	  Consumer	  (Web)	  Videos? 2	  …	  ➔	  Highlights	  2010-­‐2011	  •  Novel	  audio	  features	  for	  events	  (transients)	  and	  environments	  (textures)	  •  Release	  of	  Columbia	  Consumer	  Video	  dataset	  annotated	  via	  Amazon	  Mechanical	  Turk	  •  Best	  result	  in	  TRECVID	  2010	  	  Mul4media	  Event	  Detec4on	  evalua4on	  Event	  +	  Environment	  	  Soundtrack	  Features	  •  Conven4onal	  Bag-­‐of-­‐MFCC	  features:	  – everything	  mixed	  in	  together	  •  Can	  we	  differen4ate	  	  foreground	  and	  background?	  MFCC(20 dims)AveragePrec.SoundGroundtruthChop into10 sec blocks ∆∆2Mean One-vs-allSVMStandardizeacrosstrainsetµ(60 dims)C, γCovarianceΣ (399 samples)Foreground:	  Transient	  Features	  •  Transients	  =	  foreground	  events?	  •  Onset	  detector	  finds	  energy	  bursts	  – best	  SNR	  •  Represent	  with	  PCA	  basis	  – 300	  ms	  x	  aud	  freq	  •  “bag	  of	  transients”	  NMF	  Transient	  Features	  •  Learn	  20	  patches	  by	  	  Nonnega4ve	  Matrix	  Factoriza4on	  •  Compare	  to	  MFCC-­‐HMM	  	  •  NMF	  more	  noise-­‐robust	  – combines	  well	  Background:	  Texture	  features	  •  Characterize	  sounds	  by	  perceptually-­‐sufficient	  sta4s4cs..	  •  	  	  – verified	  by	  matched	  	  resynthesis	  •  Subband	  	  distribu4ons	  	  &	  env	  x-­‐corrs	  – Mahalanobis	  	  distance	  ...	  Texture	  Feature	  Results	  •  Test	  on	  MED	  2010	  	  development	  data	  – 10	  labels	  •  Contrasts	  in	  feature	  sets	  – correla4on	  of	  labels	  •  Perform	  ~	  same	  as	  MFCCs	  – combine	  well	  Audio	  Classifier	  EvaluaOon	  •  Inves4ga4ng	  beyond	  mAP…	  –  Accuracy,	  Mutual	  Informa4on	  Propor4on,	  Correla4on	  –  Cross-­‐corpus	  evalua4ons	  Average Precision (mean=0.397)  r u q n o D S M C L RNBasketballBaseballSoccerIceSkatingSkiingSwimmingBikingCatDogBirdGraduationBirthdayWedRecepWedCeremWedDanceMusicPerfNonMusicPerfParadeBeachPlaygroundRANDBasketballBaseballSoccerIceSkatingSkiingSwimmingBikingCatDogBirdGraduationBirthdayWedRecepWedCeremWedDanceMusicPerfNonMusicPerfParadeBeachPlaygroundRANDBasketballBaseballSoccerIceSkatingSkiingSwimmingBikingCatDogBirdGraduationBirthdayWedRecepWedCeremWedDanceMusicPerfNonMusicPerfParadeBeachPlaygroundRANDClassifiers (CCV) Balance set acc (mean=0.515)  r u q n o D S M C L RNLabels (MED)Mutual Info Prop (mean=0.090)  r u q n o D S M C L RNClassifier margin correlation coefficient  BbBsSoIcSkSwBiCaDoBiGrBdWrWcWdMpNpPaBePlRNBasketballBaseballSoccerIceSkatingSkiingSwimmingBikingCatDogBirdGraduationBirthdayWedRecepWedCeremWedDanceMusicPerfNonMusicPerfParadeBeachPlaygroundRAND00. 510.050.10.150.2−1−0.500.51MIP =I(classifier; label)H(label)Audio	  Classifier	  Results	  Browsing	  •  Customized	  version	  of	  GGobi	  links	  to	  Movie	  Player	  –  Rapid	  inves4ga4on	  of	  high-­‐dimensional	  data	  sets	  –  Each	  point	  is	  a	  video,	  colored	  by	  label	  (e.g.	  Event)	  Basketball Baseball SoccerIce Skating Skiing SwimmingBiking Cat Dog Bird GraduationBirthday Celebration Wedding Reception Wedding Ceremony Wedding DanceMusic PerformanceNon-music PerformanceParadeBeachPlayground  11	  Columbia	  Consumer	  Video	  (CCV)	  Database	  CCV	  Snapshot	  •  # videos: 9,317  –  (210 hrs in total) •  video genre –  unedited consumer videos •  video source –  YouTube.com •  average length –  80 seconds •  # defined categories –  20  •  annotation method –  Amazon Mechanical Turk 0	   100	   200	   300	   400	   500	   600	   700	   800	  music	  perf.	  non-­‐music	  perf.	  dog	  swimming	  skiing	  parade	  cat	  ice	  ska4ng	  beach	  basketball	  wedding	  dance	  bird	  playground	  soccer	  birthday	  baseball	  gradua4on	  biking	  wedding	  recep4on	  wedding	  ceremony	  12	  The trick of digging out consumer videos from YouTube:  Use default filename prefix of many digital cameras: “MVI and parade”. •  Human Action Recognition – KTH & Weizmann   •  (constrained environment)       2004-05 – Hollywood Database  •  (12 categories, movies)             2008 – UCF Database  •  (50 categories, YouTube Videos) 2010 •  Kodak Consumer Video  •  (25 classes, 1300+ videos)        2007 •  LabelMe Video  •  (many classes, 1300+ videos)    2009 •  TRECVID MED 2010  •  (3 classes, 3400+ videos)          2010 ExisOng	  Database?CCV Database Unconstrained YouTube videos Higher-level complex events More videos & better defined categories More videos & larger content variations More videos & categories Crowdsourcing:	  Amazon	  Mechanical	  Turk	   A web services API that allows developers to easily integrate human intelligence directly into their processing Task Is this a “parade” video? o Yes o No  $?.?? Internet-scale workforce What can I do for you? financial rewards 14	  MTurk:	  AnnotaOon	  Interface	  $ 0.02 Reliability of Labels: each video was assigned to four MTurk workers Human	  RecogniOon	  Performance	  16	  •  How to measure human (MTurk workers) recognition accuracy? – We manually and carefully labeled 896 videos •  Golden ground truth! •   Consolidation of the 4 sets of labels 0 0.2 0.4 0.6 0.8 1 1-vote 2-votes 3-votes 4-votes precison recall Plus additional manual filtering of 6 positive sample sets: 94% final precision Download	  - Unique YouTube Video IDs,  - Labels, - Training/Test Partition,  - Three Audio/Visual Features http://www.ee.columbia.edu/dvmm/CCV/ Fill out this … TRECVID	  MED	  2010	  •  Find	  “mul4media	  events”	  among	  1700	  videos	  •  3	  target	  event	  categories:	  Overview:	  4	  major	  components	  &	  6	  runs	  19	  Feature	  extracOon	  SIFT	  SpaOal-­‐temporal	  interest	  point	  	  MFCC	  audio	  feature	  54221	  scene,	  acOon,	  audio	  concepts	  SemanOc	  Diffusion	  with	  Contextual	  Detectors	  χ2	  	  SVM	  Classifiers	  6EMD-­‐	  SVM	  3Re-­‐Rank	  1Overview:	  overall	  performance	  20	  •  45	  systems	  by	  8	  teams	  from	  around	  the	  world	  •  Novel	  “normalized	  cost”	  metric	  •  Six	  Columbia	  systems	  scored	  best	  Roadmap	  >	  mulOple	  modaliOes	  21	  Feature	  extracOon	  SIFT	  SpaOal-­‐temporal	  interest	  point	  	  MFCC	  audio	  feature	  54221	  scene,	  acOon,	  audio	  concepts	  SemanOc	  Diffusion	  with	  Contextual	  Detectors	  χ2	  	  SVM	  Classifiers	  6EMD-­‐	  SVM	  3Re-­‐Rank	  1Three	  Feature	  ModaliOes…	  22	  •  SIFT	  (visual)	  	  – D.	  Lowe,	  IJCV	  04.	  •  STIP	  (visual)	  –  I.	  Laptev,	  IJCV	  05.	  •  MFCC	  (audio)	   …  16ms 16ms Bag-­‐of-­‐X	  RepresentaOon	  •  X = SIFT or STIP or MFCC •  Soft weighting (Jiang, Ngo and Yang, ACM CIVR 2007) Bag-­‐of-­‐SIFT	  23	  Results	  on	  Dry-­‐run	  ValidaOon	  Set	  24	  •  Measured by Average Precision (AP) •  STIP works best for event detection •  The 3 features are highly complementary! •  Should be jointly used for multimedia event detection 	   	   Assembling	  a	  shelter Ba`ng	  a	  run	  in Making	  a	  cake Mean	  AP Visual	  STIP 0.468 0.719 0.476 0.554 Visual	  SIFT 0.353 0.787 0.396 0.512 Audio	  MFCC 0.249 0.692 0.270 0.404 STIP+SIFT 0.508 0.796 0.476 0.593 STIP+SIFT+MFCC 0.533 0.873 0.493 0.633 Roadmap	  >	  temporal	  matching	  25	  Feature	  extracOon	  SIFT	  SpaOal-­‐temporal	  interest	  point	  	  MFCC	  audio	  feature	  54221	  scene,	  acOon,	  audio	  concepts	  SemanOc	  Diffusion	  with	  Contextual	  Detectors	  Re-­‐Rank	  1χ2	  	  SVM	  Classifiers	  6EMD-­‐	  SVM	  3Temporal	  Matching	  With	  EMD	  Kernel	  26	  •  Earth Mover’s Distance (EMD) •  EMD Kernel: K(P,Q)=exp-ρEMD(P,Q) Y. Rubner, C. Tomasi, L. J. Guibas, “A metric for distributions with applications to image databases”, ICCV, 1998. D. Xu, S.-F. Chang, “Video event recognition using kernel methods with multi-level temporal alignment”, PAMI, 2008. Given two frame sets P = {(p1, wp1), ... , (pm,wpm)}  and Q = {(q1, wq1), ... , (qn,wqn)} , the EMD is computed as  EMD(P, Q) = ΣiΣj fijdij / ΣiΣj fij dij is the χ2 visual feature distance of frames pi and qj. fij (weight transferred from pi and qj) is optimized by minimizing the overall transportation workload ΣiΣj fijdij Temporal	  Matching	  Results	  •  EMD is helpful for two events  –  results measured by minimal normalized cost (lower is better) 27	  5% gain Conclusions	  •  Novel	  audio	  features	  focus	  on	  foreground	  and	  background	  – Successful	  combina4ons	  •  Large-­‐scale	  annota4on	  for	  public	  data	  set	  – Columbia	  Consumer	  Video	  •  Mul4media	  Event	  Detec4on	  is	  feasible	  – Columbia	  system	  came	  top	  in	  TREC	  evalua4on	  

Joint Audio-Visual Signatures for Web Video Analysis

https://academiccommons.columbia.edu/doi/10.7916/D8VH5X01/download

Joint Audio-Visual Signatures for Web Video Analysis

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons