Presentation of video classification project

Chang, Shih-Fu

Cotton, Courtenay Valentine

Ellis, Daniel P. W.

Jiang, Wei

Jiang, Yu-Gang

Zeng, Xiaohong

English

Columbia University Academic Commons

Web Video Analysis - Dan Ellis 2010-09-14       /2311. Web Video Analysis2. Labeled Data Gathering3. Scene / Object Context4. Future WorkJoint Audio-Visual Signatures for Web Video AnalysisDan Ellis & Shih-Fu ChangWei Jiang, Courtenay Cotton, Yu-Gang Jiang, Xiaohong ZengElectrical Engineering, Columbia University, NY USAdpwe@ee.columbia.edu         http://labrosa.ee.columbia.edu/Web Video Analysis - Dan Ellis 2010-09-14       /231.  Web Video Analysis•  Web videoHuge volumePoor labelsLow quality• The goal:Automatic, efficient, human-like labeling- into categories- by events/objects2Web Video Analysis - Dan Ellis 2010-09-14       /23Conventional Approach: Global Features• Train classifiers for predefined categoriesbased on statistics of whole clipno object-level description3a video clip. . . time (e.g., color, texture)Visual FeatureAudio Feature(e.g., spectrogram)fusionstatisticalmodelingglobal feature[Chang et al. MIR 2007][Cristani et.al., TMM 2007]Web Video Analysis - Dan Ellis 2010-09-14       /23Novel Approach:Audio-Visual Atoms• Decompose aud/vid into object-like atomsstatistical models of their combinationsJoint Audio-Visual Atom discoverytime. . .visual atom. . .visual atom. . .visual atom. . .timeaudio spectrumfrequencytimefrequencyaudio atomaudio atomaudio atom4Web Video Analysis - Dan Ellis 2010-09-14       /23Challenges in Unconstrained Video• Poor qualityfocuslightingcamera motionocclusionsambient noisehandling noise• Poor A-V correlationsounds from unobserved objectssound-producing motions are slight5Web Video Analysis - Dan Ellis 2010-09-14       /23Visual Atom Formation• Point trackingsift pointslink successiveframes• + Region Segmentationcolor / texturedefine regions• Link shorter tracks across time[Jiang et.al., MM 2009]……Point Tracking for Temporal Evolution Image Segmentationfor Spatial LocalizationRegion Trackingby Point Tracking6Web Video Analysis - Dan Ellis 2010-09-14       /23Visual Atom Examples• A few examples out of 100+ for “wedding”build codebook based on appearance, shape7Web Video Analysis - Dan Ellis 2010-09-14       /23Audio Atom Formation• Extract & describe Transientsfor maximum robustness against noise• Multiscale analysis to find energy “bursts”extract 250 ms mel-spectrum windowdescribe with PCA8Web Video Analysis - Dan Ellis 2010-09-14       /23Audio Atom Results• K-means clustering to form codewords9Example patches in each cluster• Better noise resistancethan MFCCsSNR / dBAccuracyWeb Video Analysis - Dan Ellis 2010-09-14       /23Joint Audio-Visual Atoms• Consider all possible A-V combinations10audio atomaudio atom…visual atomtimeN audio atoms…M Visual Atoms… …visual atomtimecodebook learning Joint A-V CodebookM×N A-V Atoms…Web Video Analysis - Dan Ellis 2010-09-14       /23Concept Codebook Learning• by Multiple-Instance Learning (MIL)only have clip-level labels11optimal codewordfar from all negative atoms, close to at least one positive atom in positive clipspositive (+)“Wedding” Clip…a-v atom a-v atomnegative (-)Non-“Wedding” Clip…a-v atom a-v atomall negative atomsat least onepositive atomMany codewords form a codebook[Maron et al., NIPS, 1998]Web Video Analysis - Dan Ellis 2010-09-14       /23Evaluation on Consumer Video• Kodak consumer video benchmark set1358 videos (813 training)25 labels12[Loui et al. MIR 2007]ski dancing parade animal playgroundwedding sports birthday beach showWeb Video Analysis - Dan Ellis 2010-09-14       /23Example A-V Codewords• “Wedding” class13red carpetromanticmusic+black suitromanticmusic+green grassromanticmusic+white gownromanticmusic+Web Video Analysis - Dan Ellis 2010-09-14       /23Example A-V Codewords• “Parade” class14paradesoundmarching+peopleparadesoundroad+paradesoundmarching+peopleparadesoundroad+Web Video Analysis - Dan Ellis 2010-09-14       /23Example A-V Codewords• “Beach” class15beachsoundbeach+chairsandbeachsound+waterbeachsound+beachsoundpeople with+swim suitWeb Video Analysis - Dan Ellis 2010-09-14       /23Consumer Video Evaluation• A-V atoms vs. Static regionsAverage Precision on test set16Random guessStatic visual region with MILA-V atom with MIL (spatial-temporal)Web Video Analysis - Dan Ellis 2010-09-14       /23Consumer Video Evaluation• Audio / Video / bothaudio useful for many classes17Audio onlyVisual onlyVisual + Audio?Web Video Analysis - Dan Ellis 2010-09-14       /232.  Labeled Data Gathering• Learning codebooks requires labeled datanovel concept → need new labelsmore labels → better performance• Amazon “Mechanical Turk”18!Web Video Analysis - Dan Ellis 2010-09-14       /23MTurk Results• Data: YouTube raw camera uploadsbased on keyword search for 20 categories• MTurk Human Intelligence Tasks (HITs)paid $0.02 per 10s clip (~$7/hr)4 labelers/clip, finished 9,641 videos in 2 weeks19Playground,  Biking Non-music Performance, Ice SkatingWeb Video Analysis - Dan Ellis 2010-09-14       /233.  Scene / Object Context• How to identify events in video?not objects, not locationse.g. “people kissing”• Traditional approach:get low level features for large training setstatistical classifier• Our approachuse specialized mid-leveldetectors (faces, cars)learn context, relationshipse.g. “kissing” = 2 faces moving together20!!"#$%&'()*&)+,-&.)/,01),/2&30)45-)63&5&)3,50&706)8,%)&9&50)+,-&."5#:)*&)8"%60)-&0&30),/2&306)6$31)46);&%6,5)$6"5#)6040&<,8<01&<4%0),/2&30)-&0&30,%6)=%"#10>?)45-)3.466"8@)9"-&,)63&5&6)$6"5#);%&<0%4"5&-)63&5&)+,-&.6)=.&80>:)A5)4.#,%"01+)"6);%,;,6&-)0,);%&-"30)&9&50<,/2&30<63&5&)%&.40",561";)8%,+)4)6+4..)5$+/&%),8)0%4"5"5#)64+;.&6?)B1"31)"6)8"54..@)$6&-)8,%)8"5-"5#)C"66"5#)"5)5&B)9"-&,6:)Event [Jiang, Li, Chang, TSCVT 2011]Web Video Analysis - Dan Ellis 2010-09-14       /23Action-Scene-Object• Identify relevant objects, scenes from a few training examples (~ 10)• Learn relationships for actionaccuracy much better than raw classifier213!!!"#$%&$'()*+&%,-(%#"#.#!,/-$",0"(&$%,0&$'1(!2,$,(3,2-'$&"&-$(#$%(-.4,2"(%,",2"&-$5-%,6&$'(#2"&-$7!2,$,7-.4,2"(3,6#"&-$!8&*#2"&-$ !2,$,-.4,2"92"&-$(:-%,6&$'(;&"8(2-$",0"<&:&6#3(#2"&-$!=>>6&$,(2-$",0"(&$%,0&$'!!*,3!-$28#&3*,3!-$28#&328#&3*,3!-$728#&37#;#?->>&2, .,%3--:2#3*,3!-$28#&328#&3*,3!-$*,3!-$28#&3*,3!-$28#&3@@""#"$"#%"#%$"#&"#&$'()*+*,-,./0)1234)+5635())78+(0).3-9,.:;3<=,>,<?#4))7@AA,:3432.+1(+<.B;)'""#"$"#%"#%$"#&'()*+*,-,./0)1234)+5635())78+(0).3 -9,.:;3<=,>,<?#4))7@AA,:3432.+1(+<.B;)'""#"$"#%"#%$"#&'()*+*,-,./0)1234)+5635())78+(0).3-9,.:;3<=,>,<?#4))7@AA,:3432.+1(+<.B;)'Fig. 2. An verview of our context-based action retrieval framework. Given a few example video clips of an action (standing up), we first automatically minea number of pseudo-negative samples. We then extract object and scene contexts and predict their relationships to the action. The learned action-scene-objectrelationship is finally utilized to incorporate multiple contextual cues for recognizing similar actions.script mining method in [3] to collect labeled samples, as itis particularly designed for movies and may not be appliedfor other types of action videos. While in the experiments theHollywood2 dataset is adopted for the ease of performanceevaluation, we do not want to limit our approach to anyparticular type of data.B. Obtaining Action Context1) Scene Recognition: Background scene setting is animportant source of context for understanding human actions.We adopt ten scene classes defined in [5] (see Fig. 1-Left forclass names), and the scene models are learned using one-against-all strategy by Support Vector Machines (SVM). Weuse the popular χ2 Gaussian kernel. The effectiveness of thiskernel has been validated in many visual recognition tasks.To combine the two bag-of-features representations based onSIFT and HoG-HoF, we train separate classifiers and simplyaverage their probability predictions. From our evaluation, ourscene models perform very similar to that in [5].2) Object-level Action Atoms: Besides the backgroundscene, another useful contextual cue is the types of objectsand their interactions in action videos. We therefore employa state-of-the-art generic object detector [37], with modelstrained for locating person, car, and chair. Since our aim is tolearn human actions, person is obviously the most importantobject of interest, while other objects such as car and chairare helpful for identifying actions involving object interactions(e.g., person getting out of car and standing up). Objectdetectors other than the three can be easily deployed in ourframework without any significant modification.The output of the detectors is a set of object boundingboxes over static video frames. Since temporal informationis not considered during detection, the results are not alwaysconsistent across nearby frames – there are many false alarmsand miss-detections. To alleviate the effect of this issue, we!"#$%& !"#$%&!"#$%& !"#$%&!"#$%&!"#$%&'(#'(#!"#$%&!"#$%& '(#!"#$%&)'(#)'*%$"#!"#$%&'(#!"#$%&'(#!"#$%&)'(#)(+(,!"#$%&!"#$%&!"#$%&!"#$%&!"#$%&)!"#$%&)'*%$"# !"#$%&)'-(.#)(+(,!"#$%&'-(.#'-(.#!"#$%&'-(.#!"#$%&'-(.#!"#$%&'-(.#!"#$%&)'-(.#)'*%$"#%&"/!"#$%& 0+%/!"%!*" 0-#""/%#/1%#"/!"%!*" '(#'-(.#'-(.#Fig. 3. Object-level action atoms. The first five indicate object presenceand/or count, and the rest describes object interactions.track the detected objects based on their spatial locationsand bounding box sizes (bounding box overlap must exceed40%), and discard the isolated detections without any trackedassociation within a temporal window. The window size isempirically set as 15 frames (0.5 seconds).Based on the detection, here we define ten object-levelaction atoms, including single object, multiple objects, as wellas object interactions (with varying spatial distances alongtime). Fig. 3 gives an exemplar for each of the atoms. Weexpect that knowing the presence of these atoms, though noisy,will be helpful for recognizing human actions (e.g., person-person-closer for action kissing). To characterize the objectinteractions, we compute average spatial distance betweendifferent types of detected objects (or among all detectedpeople for person-person-closer). The average distance iscompared to that in the next successive frame in order toWeb Video Analysis - Dan Ellis 2010-09-14       /234.  Future Work• Existing joint Audio-Visual atoms are based on simple co-occurrenceno temporal structure• Synchrony?too hard to detect in web video• Causality?e.g. simple ordering“Causal Audio-Video Atoms”CAVAs22! Figure 3: Generating Causal Audio-Visual Atom (CAVA) Web Video Analysis - Dan Ellis 2010-09-14       /23Summary• Web video analysisdesperate need for automatic analysismust be in terms of objects, scenes, actions• Joint Audio-Visual Atomsobject-related codebooks for audio, videoMIL of all possible combinations to find cues• Labeled DataMechanical Turk quickly labels web video examples• Context-Based Action Detectionuses mature existing object and scene detectors• Better “Causal Audio-Visual Atoms”23

Joint Audio-Visual Signatures for Web Video Analysis

https://academiccommons.columbia.edu/doi/10.7916/D8417561/download

Joint Audio-Visual Signatures for Web Video Analysis

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons