Comunicación presentada en MIREX 2008 - Music Information Retrieval Evaluation eXchange, MIREX Genre Classification Contest., Philadelphia, Pennsylvania, USA (2008).The novel approach of combining audio and symbolic features for music classification from audio enhanced previous audio-only based results in MIREX 2007. We extended the approach by including temporal audio features, enhancing the polyphonic audio to MIDI transcription system and including an extended set of symbolic features. Recent research in music genre classification hints at a glass ceiling being reached using timbral audio features.This work is supported by the Austrian Academic Exchange Service (ÖAD) through the IMPACT project, the Spanish PROSEMUS project with code TIN2006-14932-C02 and the Pascal Network of Excellence

Iñesta, José M.

Lidy, Thomas

Pertusa, Antonio

Ponce de León Amador, Pedro José

Rauber, Andreas

Audio music classification using a combination of spectral, timbral, rhythmic, temporal and symbolic features

Repositorio Institucional de la Universidad de Alicante

MIREX 2008AUDIO MUSIC CLASSIFICATION USING A COMBINATION OF SPECTRAL,TIMBRAL, RHYTHMIC, TEMPORAL AND SYMBOLIC FEATUREST. Lidy, A. RauberVienna University of Technology, AustriaDepartment of Software Technologyand Interactive SystemsA. Pertusa, P. J. Ponce de Leo´n, J. M. In˜estaUniversity of Alicante, SpainDepartamento de Lenguajes ySistemas Informa´ticosABSTRACTThe novel approach of combining audio and symbolic fea-tures for music classification from audio enhanced previousaudio-only based results in MIREX 2007. We extended theapproach by including temporal audio features, enhancingthe polyphonic audio to MIDI transcription system and in-cluding an extended set of symbolic features. Recent re-search in music genre classification hints at a glass ceilingbeing reached using timbral audio features.1 INTRODUCTIONClassification of music by genre, artist or mood are impor-tant tasks for retrieval and organization of music databases.Traditionally the research domain of music classificationwas divided into the audio and symbolic music analysisand retrieval domains. Our work is aimed at combining ap-proaches from both directions that have proved their reliabil-ity in their respective domains. We are combining spectrum-based audio feature extractors, that include aspects such asrhythm, timbre and temporal evolution of signals on variouscritical frequency bands, with symbolic descriptors, basedon note onsets and statistics, using a polyphonic transcrip-tion system as an intermediate step. These features are com-plementary; a score can provide very valuable information,but audio features (e.g., the timbral information) are alsovery important for classification, e.g. into various genres.To extract symbolic descriptors from an audio signal it isnecessary to first employ a transcription system in order todetect the notes stored in the signal. Transcription systemshave been investigated previously but a well-performing so-lution for polyphonic music and a multitude of genres hasnot yet been found. Though these systems might not be ina final state for solving the transcription problem, our hy-pothesis is that they are able to augment the performance ofmusic classification by introducing features on the symboliclevel.The overall scheme of our proposed genre classificationsystem is shown in Figure 1. It processes an audio file in twoways to predict its genre. While in the first branch, the au-dio feature extraction methods described in Section 2.1 areapplied directly to the audio signal data, there is an interme-diate step in the second branch. A polyphonic transcriptionsystem, described in Section 2.2.1, converts the audio infor-mation into a symbolic notation (i.e. MIDI files). Then, asymbolic feature extractor is applied on the resulting repre-sentation, providing a set of symbolic descriptors as output.The audio and symbolic features extracted from the musicserve as combined input to a classifier.The basic system is outlined and described in more de-tail in [2]. We extended the approach by including temporalaudio features, enhancing the polyphonic transcription sys-tem and including an extended set of symbolic features, asoutlined in the following section.2 SYSTEM DESCRIPTION2.1 Audio Feature ExtractionAll the following descriptors are extracted from a spectralrepresentation of 6 sec. segments in the audio signal.While in full length songs, the number of segments variesand can be controlled using a ’step width’ parameter, ina 30-second audio clip, usually 5 segments are extracted.Rhythm Patterns and Rhythm Histograms are summarizedusing the median over the 5 segments, Statistical SpectrumDescriptors are summarized computing the mean. For Tem-poral Rhythm Histograms and Temporal Statistical Spec-trum Descriptors statistics that measure variation over timeFigure 1. General framework of the system(i.e. over the 5 segments) are computed. Note that in con-trast to MIREX 2007 we did not include Onset features inthis submission (due to a change in implementation).2.1.1 Rhythm Pattern (RP)The feature extraction process for a Rhythm Pattern [4, 1]is composed of two stages. First, the specific loudness sen-sation on 24 critical frequency bands is computed, by usinga Short Time FFT, grouping the resulting frequency bandsto the Bark scale, applying spreading functions to accountfor masking effects and successive transformation into theDecibel, Phon and Sone scales. This results in a psycho-acoustically modified Sonogram representation that reflectshuman loudness sensation. In the second step, a discreteFourier transform is applied to this Sonogram, resulting in a(time-invariant) spectrum of loudness amplitude modulationper modulation frequency for each individual critical band.After additional weighting and smoothing steps, a RhythmPattern exhibits magnitude of modulation for 60 modula-tion frequencies (between 0.17 and 10 Hz) on the 24 criticalbands. Note that when using 22khz audio, the number ofcritical bands is reduced to 20 and the final Rhythm Patternhas 1200 dimensions. For details refer to [4, 1].2.1.2 Rhythm Histogram (RH)A Rhythm Histogram (RH) aggregates the modulation am-plitude values of the individual critical bands computed in aRhythm Pattern and is thus a lower-dimensional descriptorfor general rhythmic characteristics in a piece of audio [1].A modulation amplitude spectrum for critical bands accord-ing to the Bark scale is calculated, as for Rhythm Patterns.Subsequently, the magnitudes of each modulation frequencybin of all critical bands are summed up to a histogram, ex-hibiting the magnitude of modulation for 60 modulation fre-quencies between 0.17 and 10 Hz.2.1.3 Temporal Rhythm Histogram (TRH)Statistical measures (mean, median, variance, skewness,kurtosis, min and max) are computed over the individualRhythm Histograms extracted from various segments in apiece of audio. Thus, change and variation of rhythmic as-pects in time are captured by this descriptor.2.1.4 Statistical Spectrum Descriptor (SSD)In the first part of the algorithm for computation of a Statis-tical Spectrum Descriptor (SSD) the specific loudness sen-sation is computed on 24 Bark-scale bands, equally as fora Rhythm Pattern. Subsequently, the mean, median, vari-ance, skewness, kurtosis, min- and max-value are calculatedfor each individual critical band. These features computedfor the 24 bands constitute a Statistical Spectrum Descrip-tor. SSDs describe fluctuations on the critical bands and areable to capture additional timbral information compared toa Rhythm Pattern, yet at a much lower dimension of the fea-ture space, as shown in the evaluation in [1].2.1.5 Temporal Statistical Spectrum Descriptor (TSSD)Statistical measures (mean, median, variance, skewness,kurtosis, min and max) are computed over the individualStatistical Spectrum Descriptors extracted from the varioussegments of a piece of audio. This captures timbral varia-tions and changes over time in the spectrum on the individ-ual critical frequency bands.2.1.6 Modulation Frequency Variance Descriptor (MVD)This descriptor measures variations over the critical fre-quency bands for a specific modulation frequency (derivedfrom a Rhythm Pattern). Consider a Rhythm Pattern, i.e.a matrix representing the amplitudes of 60 modulation fre-quencies on 24 critical bands: The MVD vector is computedby taking statistics (mean, median, variance, skewness, kur-tosis, min and max) for one modulation frequency over the24 (resp. 20) bands. A vector is computed for each of the 60modulation frequencies. The MVD descriptor for an audiofile is computed from the mean over the multiple MVDs ofits segemens.2.2 Symbolic Feature Extraction2.2.1 Transcription SystemTo complement the audio features with symbolic featureswe developed a polyphonic transcription system to extractthe notes. This system converts the audio signal into a MIDIfile that will later be analyzed to extract the symbolic de-scriptors. It does not consider rhythm, only pitches and notedurations are extracted. Therefore, the transcription systemconverts a mono audio file sampled at 22 kHz into a se-quence of notes. The transcription system is described indetail in [2], the latest version is presented at the MIREX2008 Multiple Fundamental Frequency Estimation & Track-ing task with an abstract describing the system.2.2.2 Symbolic FeaturesA set of 53 symbolic descriptors was extracted from thetranscribed notes. This set is based on the features de-scribed in [3], that yielded good results for monophonicclassical/jazz classification, and on the symbolic featuresdescribed in [6], used for melody track selection in MIDIfiles. The number of notes, number of significant silences,and the number of non-significant silences were computed.The occupation rate (sounding notes periods with respectto song length) and polyphony rate (proportion of sound-ing note periods with more than one note active simultane-ously) were also computed. Note pitches, pitch intervals,note durations, silence durations, Inter Onset Intervals (IOI)and non-diatonic notes were also analyzed (for the latter, thesong key is guessed using the algorithm described in [5]).Each one of this properties is described by their highest andlowest values, their range, average, relative average, stan-dard deviation, and a normality estimation. The total num-ber of IOI was also taken into account, as the number ofdistinct pitch intervals, the most repeated pitch interval, thesum of all note durations and an estimation of the numberof syncopations in the song, completing the symbolic fea-ture set.3 CLASSIFICATION3.1 Classification SetupWith the availability of multiple feature sets as a source ofmusic description, and potentially also multiple classifiers,there are several alternatives of how to design a music clas-sification system. The option we chose is to merge the dif-ferent feature sets and provide the combined set as input toa common classifier that receives an extended set of fea-ture attributes on which it bases its classification decision(c.f. Figure 1). For our experiments we chose linear Sup-port Vector Machines. We used the SMO implementation ofthe Weka machine learning software [7] with pairwise clas-sification and the default Weka parameters (complexity pa-rameter C = 1.0). We investigated the performance of thefeature sets individually in advance and then decided whichfeature sets to combine.3.2 Feature CombinationsVarious combinations of different feature sets achieve diver-gent results on different music test collections. Therefore,we included 4 algorithm variants in our submission, in orderto observe the differing performances of different feature setcombinations. Moreover, comparing the variants is impor-tant to gain insight in the usefulness of symbolic features foraudio music classification. The 4 variants are based on thefollowing sets of features (refer to Section 2):1. RH+SSD (audio only)2. RP+MVD+SSD (audio only)3. RP+SSD+TRH+Symbolic4. RP+RH+TRH+SSD+TSSD+MVD+Symbolic4 ACKNOWLEDGMENTSThis work is supported by the Austrian Academic ExchangeService (O¨AD) through the IMPACT project and the Span-ish PROSEMUS project with code TIN2006-14932-C02.5 REFERENCES[1] T. Lidy and A. Rauber. Evaluation of feature extractorsand psycho-acoustic transformations for music genreclassification. In Proc. ISMIR, pages 34–41, London,UK, September 11-15 2005.[2] T. Lidy, A. Rauber, A. Pertusa, and J.M. In˜esta. Im-proving genre classification by combination of audioand symbolic descriptors using a transcription system.In Proc. ISMIR, Vienna, Austria, 2007.[3] P. J. Ponce de Leo´n and J. M. In˜esta. A pattern recogni-tion approach for music style identification using shal-low statistical descriptors. IEEE Trans. on Systems Manand Cybernetics C, 37(2):248–257, 2007.[4] A. Rauber, E. Pampalk, and D. Merkl. The SOM-enhanced JukeBox: Organization and visualization ofmusic collections based on perceptual models. Journalof New Music Research, 32(2):193–210, June 2003.[5] D. Rizo, J.M. In˜esta, and P.J. Ponce de Leo´n. Tree modelof symbolic music for tonality guessing. In Proc. of theIASTED Int. Conf. on Artificial Intelligence and Appli-cations, AIA 2006, pages 299–304, Innsbruck, Austria,2006. IASTED, Acta Press. ISBN 0-88986-404-7.[6] D. Rizo, P.J. Ponce de Leo´n, C. Pe´rez-Sancho, A. Per-tusa, and J.M. In˜esta. A pattern recognition approachfor melody track selection in midi files. In Proc. ISMIR,pages 61–66, Victoria, Canada, 2006.[7] I.H. Witten and E. Frank. Data Mining: Practical ma-chine learning tools and techniques. Morgan Kaufmann,San Francisco, 2nd edition, 2005.

Audio music classification using a combination of spectral, timbral, rhythmic, temporal and symbolic features

Abstract

Similar works

Full text

Available Versions

RUA

Repositorio Institucional de la Universidad de Alicante