An overview of the current status and applications of semantic audio analysis

Ellis, Daniel P. W.

English

Columbia University Academic Commons

 Dan Ellis Semantic Audio Analysis (1 of 19) 2003-10-13  Semantic Audio Analysis Semantic Audio AnalysisOrganizing Sound MixturesApplications for Audio SemanticsOpen Questions1234 Dan Ellis  <dpwe@ee.columbia.edu>Laboratory for Recognition and Organization of Speech and Audio(Lab ROSA )Columbia University, New Yorkhttp://labrosa.ee.columbia.edu/ Dan Ellis Semantic Audio Analysis (2 of 19) 2003-10-13  Semantic Audio Analysis • Audio Semantics= what is the meaning / message? - “AI complete”? • “Semantics” is broad! - used for the-stuff-we-can’t-do-yet • How about speech recognition? IT'S NICE TO HAVE THIS FRIDAY WAS IN DOLLARS IN THE BOMBING RAIDS WAS CLOSING THOUSAND TO FOUR GUNMEN CAUSING CONDITIONING SAID THE STOCK ROSE SMOKERS FROM THE TWENTY NINE NINETY FIVE PLUS THREE ON THE EXPERT TECHNICIANS EXPECTED TO REACH - no good even if it worked!10 2 4 6 8 10 12 time/sfrq/Hz02000100030004000level / dB-60-40-200 Dan Ellis Semantic Audio Analysis (3 of 19) 2003-10-13  Towards Semantic Audio Analysis • What do we want from SAA? - describe sound in human-recognizable terms- “automatic subtitles for real life”? • Key step in speech recognition is classification - convert continuous signal into discrete classes- need a rich, application-dependent vocabulary- hierarchy of pattern recognition • Listeners perceive sound sources - If SAA primitives are to be subjective percepts...- ...source segregation is the first problem?[thunder]S: I THINK THEWEATHER'S CHANGING Dan Ellis Semantic Audio Analysis (4 of 19) 2003-10-13  SAA Applications • Subjective descriptions are the ultimate sound representation - data compression: “Radio ad for AARCO”- signal enhancement: “... without noise”- modification: “.. woman’s voice ..”- .. needs both analysis and synthesis? • Sound understanding useful for: - indexing/retrieval- robots- prostheses?0 2 4 6 8 10 12 time/sfrq/Hz02000100030004000level / dB-60-40-200 Dan Ellis Semantic Audio Analysis (5 of 19) 2003-10-13  Outline Semantic Audio Analysis Organizing Sound Mixtures - Human Auditory Scene Analysis- Organizing Mixtures by Computer Applications for Audio SemanticsOpen Questions1234 Dan Ellis Semantic Audio Analysis (6 of 19) 2003-10-13  Auditory Scene Analysis   (Bregman 1990) • How do people analyze sound mixtures? - break mixture into small elements (in time-freq)- elements are grouped in to sources using cues- sources have aggregate attributes • Elements + attributes - common onset ( = dependent origins)- periodicity ( = single process)- + spatial cues etc. + familiarity, context ...2time / sfreq / Hz0 1 2 3 4 5 6 7 8 902000400060008000 Dan Ellis Semantic Audio Analysis (7 of 19) 2003-10-13  Computational Auditory Scene Analysis:The Representational Approach (Cooke & Brown 1993) • Direct implementation of psych. theory - ‘bottom-up’ processing- uses common onset & periodicity cues • Able to extract voiced speech:inputmixturesignalfeatures(maps)discreteobjectsFront end ObjectformationGroupingrulesSourcegroupsonsetperiodfrq.modtimefreq0.2 0.4 0.6 0.8 1.0 time/s1001502003004006001000150020003000frq/Hzbrn1h.aif0.2 0.4 0.6 0.8 1.0 time/s1001502003004006001000150020003000frq/Hzbrn1h.fi.aif Dan Ellis Semantic Audio Analysis (8 of 19) 2003-10-13  Approaches to handling sound mixtures • Separate signals, then recognize - Computational Auditory Scene Analysis (CASA), Independent Component Analysis- nice, if you can make it work • Recognize combined signal - ‘multicondition training’- combinatorics seem daunting • Recognize with parallel models - optimal inference from full joint state-space- or: skip obscured fragments, infer from higher-level context- or do both: missing-data recognitionp O x y, ,( ) p x y, O( )Æ Dan Ellis Semantic Audio Analysis (9 of 19) 2003-10-13  Multi-source decoding (Barker, Cooke & Ellis 2003) • Missing Data recognizes from a subset;Can search for more than one source• Mutually-dependent data masks• Use e.g. CASA features to propose masks - locally coherent regions • Issues in models, representations, inference...Y(t)S1(t)q1(t)S2(t)q2(t) Dan Ellis Semantic Audio Analysis (10 of 19) 2003-10-13  Outline Semantic Audio AnalysisOrganizing Sound MixturesApplications for Audio Semantics - Meeting recordings- Audio diary analysis- Semantics of musical signals Open Questions1234 Dan Ellis Semantic Audio Analysis (11 of 19) 2003-10-13  The Meeting Recorder Project (with ICSI, UW, IDIAP, SRI, Sheffield) • Microphones in conventional meetings - for summarization / retrieval / behavior analysis- informal, overlapped speech ( Æ  ASR...) • Behavioral: Look for patterns of speaker turns33:8:10:9:7:5:2:1:0 5 10 15 20 25 30 35 40 45 50 55 60mr04: Hand-marked speaker turns vs. time + auto/manual boundariesParticipanttime/min Dan Ellis Semantic Audio Analysis (12 of 19) 2003-10-13  Personal Audio: The Listening Machine • Smart PDA records everything you hear• Only useful if we have index, summaries - semantic descriptions (real time?) • Features appropriate for 1 minute segments... - Bark band variance, spectral entropy, ...50 100 150 200 25002414:30 15:00 15:30 16:00 16:30 17:00 17:30 18:00 18:3051015freq / Hzfreq / Barktime / minclock time Dan Ellis Semantic Audio Analysis (13 of 19) 2003-10-13  Music Information Retrieval (Berenzweig & Ellis 2003) • Apply search concepts to music? - “musical Google” – beat human annotation?- application: finding new music • Construct music space where near = “similar” Dan Ellis Semantic Audio Analysis (14 of 19) 2003-10-13  Outline Semantic Audio AnalysisOrganizing Sound MixturesApplications for Audio SemanticsOpen Questions1234 Dan Ellis Semantic Audio Analysis (15 of 19) 2003-10-13  Open Questions • Semantics - What are the abstract perceptual attributes of a sound?  How can we describe them? • Mixtures - How do people organize sound mixtures into separate source percepts?- How can we represent generic sound knowledge? • Applications - Search/retrieval:  What terminology is  most natural for users querying sound databases?- What is the best balance between machine and human listening?- What are the problems for which machine listening can be most useful?4 Dan Ellis Semantic Audio Analysis (16 of 19) 2003-10-13  Extra slides Dan Ellis Semantic Audio Analysis (17 of 19) 2003-10-13  Missing Data Recognition (Barker, Cooke & Ellis ’03) • Can evaluate speech models  p ( x | m )  over a subset of dimensions  x k • Hence, missing data recognition:• Fit model and segregation given obs’n:xkxuyp(xk,xu)p(xk|xu<y ) p(xk )p xk m( ) p xk xu, m( ) xudÚ=P(x1 | q) P(x | q) = · P(x2 | q) · P(x3 | q) · P(x4 | q) · P(x5 | q) · P(x6 | q) Present data masktime dimension .. but need segregation maskP M S Y,( ) P M( ) P X M( ) P X Y S,( )P X( )------------------------◊ XdÚ P S Y( )◊=Dan Ellis Semantic Audio Analysis (18 of 19) 2003-10-13 What a speech HMM contains• Markov model structure: states + transitions• A generative model- but not a good speech generator!- only meant for inference of p(X|M)KAS0.8 0.180.90.10.020.90.1T E0.80.2 M'1OKAS0.8 0.050.90.10.150.90.1T E0.80.2OModel M'201234freq / kHz10 20 30 40 501020304050State models (means) State Transition Probabilities0 1 2 3 4 501234freq / kHztime / secDan Ellis Semantic Audio Analysis (19 of 19) 2003-10-13 Music similarity from Anchor space• A classifier trained for one artist (or genre) will respond partially to a similar artist• Each artist evokes a particular pattern of responses over a set of classifiers• We can treat these classifier outputs as a new feature space in which to estimate similarity• “Anchor space” reflects subjective qualities?n-dimensionalvector in "AnchorSpace"AnchorAnchorAnchorAudioInput(Class j)p(a1|x)p(a2|x)p(an|x)GMMModelingConversion to Anchorspacen-dimensionalvector in "AnchorSpace"AnchorAnchorAnchorAudioInput(Class i)p(a1|x)p(a2|x)p(an|x)GMMModelingConversion to AnchorspaceSimilarityComputationKL-d, EMD, etc.

Semantic Audio Analysis

https://academiccommons.columbia.edu/doi/10.7916/D8MW2RBX/download

Semantic Audio Analysis

Abstract

Similar works

Full text

Available Versions

Columbia University Academic Commons