Application of missing feature theory to the recognition of musical instruments in polyphonic audio

A system for musical instrument recognition based on a Gaussian Mixture Model (GMM) classifier is introduced. To enable instrument recognition when more than one sound is present at the same time, ideas from missing feature theory are incorporated. Specifically, frequency regions that are dominated by energy from an interfering tone are marked as unreliable and excluded from the classification process. The approach has been evaluated on clean and noisy monophonic recordings, and on combinations of two instrument sounds. These included random chords made from two isolated notes and combinations of two realistic phrases taken from commercially available compact discs. Classification results were generally good, not only when the decision between reliable and unreliable features was based on the knowledge of the clean signal, but also when it was solely based on the pitch and harmonic overtone series of the interfering sound

Large scale musical instrument identification

In this paper, automatic musical instrument identification using a variety of classifiers is addressed. Experiments are performed on a large set of recordings that stem from 20 instrument classes. Several features from general audio data classification applications as well as MPEG-7 descriptors are measured for 1000 recordings. Branch-and-bound feature selection is applied in order to select the most discriminating features for instrument classification. The first classifier is based on non-negative matrix factorization (NMF) techniques, where training is performed for each audio class individually. A novel NMF testing method is proposed, where each recording is projected onto several training matrices, which have been Gram-Schmidt orthogonalized. Several NMF variants are utilized besides the standard NMF method, such as the local NMF and the sparse NMF. In addition, 3-layered multilayer perceptrons, normalized Gaussian radial basis function networks, and support vector machines employing a polynomial kernel have also been tested as classifiers. The classification accuracy is high, ranging between 88.7% to 95.3%, outperforming the state-of-the-art techniques tested in the aforementioned experiment

Multi-label Ferns for Efficient Recognition of Musical Instruments in Recordings

In this paper we introduce multi-label ferns, and apply this technique for automatic classification of musical instruments in audio recordings. We compare the performance of our proposed method to a set of binary random ferns, using jazz recordings as input data. Our main result is obtaining much faster classification and higher F-score. We also achieve substantial reduction of the model size

Robust and efficient approach to feature selection with machine learning

Most statistical analyses or modelling studies must deal with the discrepancy between the measured aspects of analysed phenomenona and their true nature. Hence, they are often preceded by a step of altering the data representation into somehow optimal for the following methods.This thesis deals with feature selection, a narrow yet important subset of representation altering methodologies.Feature selection is applied to an information system, i.e., data existing in a tabular form, as a group of objects characterised by values of some set of attributes (also called features or variables), and is defined as a process of finding a strict subset of them which fulfills some criterion.There are two essential classes of feature selection methods: minimal optimal, which aim to find the smallest subset of features that optimise accuracy of certain modelling methods, and all relevant, which aim to find the entire set of features potentially usable for modelling. The first class is mostly used in practice, as it adheres to a well known optimisation problem and has a direct connection to the final model performance. However, I argue that there exists a wide and significant class of applications in which only all relevant approaches may yield usable results, while minimal optimal methods are not only ineffective but even can lead to wrong conclusions.Moreover, all relevant class substantially overlaps with the set of actual research problems in which feature selection is an important result on its own, sometimes even more important than the finally resulting black-box model. In particular this applies to the p>>n problems, i.e., those for which the number of attributes is large and substantially exceeds the number of objects; for instance, such data is produced by high-throughput biological experiments which currently serve as the most powerful tool of molecular biology and a fundament of the arising individualised medicine.In the main part of the thesis I present Boruta, a heuristic, all relevant feature selection method. It is based on the concept of shadows, by-design random attributes incorporated into the information system as a reference for the relevance of original features in the context of whole structure of the analysed data. The variable importance on its own is assessed using the Random Forest method, a popular ensemble classifier.As the performance of the Boruta method turns out insatisfactory for some important applications, the following chapters of the thesis are devoted to Random Ferns, an ensemble classifier with the structure similar to Random Forest, but of a substantially higher computational efficiency. In the thesis, I propose a substantial generalisation of this method, capable of training on generic data and calculating feature importance scores.Finally, I assess both the Boruta method and its Random Ferns-based derivative on a series of p>>n problems of a biological origin. In particular, I focus on the stability of feature selection; I propose a novel methodology based on bootstrap and self-consistency. The results I obtain empirically confirm the validity of aforementioned effects characteristic to minimal optimal selection, as well as the efficiency of proposed heuristics for all relevant selection.The thesis is completed with a study of the applicability of Random Ferns in musical information retrieval, showing the usefulness of this method in other contexts and proposing its generalisation for multi-label classification problems.W większości zagadnień statystycznego modelowania istnieje problem niedostosowania zebranych danych do natury badanego zjawiska; co za tym idzie, analiza danych jest zazwyczaj poprzedzona zmianą ich surowej formy w optymalną dla dalej stosowanych metod.W rozprawie zajmuję się selekcją cech, jedną z klas zabiegów zmiany formy danych. Dotyczy ona systemów informacyjnych, czyli danych dających się przedstawić w formie tabelarycznej jako zbiór obiektów opisanych przez wartości zbioru atrybutów (nazywanych też cechami), oraz jest zdefiniowana jako proces wydzielenia w jakimś sensie optymalnego podzbioru atrybutów.Wyróżnia się dwie zasadnicze grupy metod selekcji cech: poszukujących możliwie małego podzbioru cech zapewniającego możliwie dobrą dokładność jakiejś metody modelowania (minimal optimal) oraz poszukujących podzbioru wszystkich cech, które niosą istotną informację i przez to są potencjalnie użyteczne dla jakiejś metody modelowania (all relevant). Tradycyjnie stosuje się prawie wyłącznie metody minimal optimal, sprowadzają się one bowiem w prosty sposób do znanego problemu optymalizacji i mają bezpośredni związek z efektywnością finalnego modelu. W rozprawie argumentuję jednak, że istnieje szeroka i istotna klasa problemów, w których tylko metody all relevant pozwalają uzyskać użyteczne wyniki, a metody minimal optimal są nie tylko nieefektywne ale często prowadzą do mylnych wniosków. Co więcej, wspomniana klasa pokrywa się też w dużej mierze ze zbiorem faktycznych problemów w których selekcja cech jest sama w sobie użytecznym wynikiem, nierzadko ważniejszym nawet od uzyskanego modelu. W szczególności chodzi tu o zbiory klasy p>>n, to jest takie w których liczba atrybutów w~systemie informacyjnym jest duża i znacząco przekracza liczbę obiektów; dane takie powszechnie występują chociażby w wysokoprzepustowych badaniach biologicznych, będących obecnie najpotężniejszym narzędziem analitycznym biologii molekularnej jak i fundamentem rodzącej się zindywidualizowanej medycyny.W zasadniczej części rozprawy prezentuję metodę Boruta, heurystyczną metodę selekcji zmiennych. Jest ona oparta o koncepcję rozszerzania systemu informacyjnego o cienie, z definicji nieistotne atrybuty wytworzone z oryginalnych cech przez losową permutację wartości, które są wykorzystywane jako odniesienie dla oceny istotności oryginalnych atrybutów w kontekście pełnej struktury analizowanych danych. Do oceny ważności cech metoda wykorzystuje algorytm lasu losowego (Random Forest), popularny klasyfikator zespołowy.Ponieważ wydajność obliczeniowa metody Boruta może być niewystarczająca dla pewnych istotnych zastosowań, w dalszej części rozprawy zajmuję się algorytmem paproci losowych, klasyfikatorem zespołowym zbliżonym strukturą do algorytmu lasu losowego, lecz oferującym znacząco lepszą wydajność obliczeniową. Proponuję uogólnienie tej metody, zdolne do treningu na generycznych systemach informacyjnych oraz do obliczania miary ważności atrybutów.Zarówno metodę Boruta jak i jej modyfikację wykorzystującą paprocie losowe poddaję w rozprawie wyczerpującej analizie na szeregu zbiorów klasy p>>n pochodzenia biologicznego. W szczególności rozważam tu stabilność selekcji; w tym celu formułuję nową metodę oceny opartą o podejście resamplingowe i samozgodność wyników. Wyniki przeprowadzonych eksperymentów potwierdzają empirycznie zasadność wspomnianych wcześniej problemów związanych z selekcją minimal optimal, jak również zasadność przyjętych heurystyk dla selekcji all relevant.Rozprawę dopełnia studium stosowalności algorytmu paproci losowych w problemie rozpoznawania instrumentów muzycznych w nagraniach, ilustrujące przydatność tej metody w innych kontekstach i proponujące jej uogólnienie na klasyfikację wieloetykietową

A Geometric Approach to Pattern Matching in Polyphonic Music

The music pattern matching problem involves finding matches of a small fragment of music called the "pattern" into a larger body of music called the "score". We represent music as a series of horizontal line segments in the plane, and reformulate the problem as finding the best translation of a small set of horizontal line segments into a larger set of horizontal line segments. We present an efficient algorithm that can handle general weight models that measure the musical quality of a match of the pattern into the score, allowing for approximate pattern matching. We give an algorithm with running time O(nm(d + log m)), where n is the size of the score, m is the size of the pattern, and d is the size of the discrete set of musical pitches used. Our algorithm compares favourably to previous approaches to the music pattern matching problem. We also demonstrate that this geometric formulation of the music pattern matching problem is unlikely to have a significantly faster algorithm since it is at least as hard as 3SUM, a basic problem that is conjectured to have no subquadratic algorithm. Lastly, we present experiments to show how our algorithm can find musically sensible variations of a theme, as well as polyphonic musical patterns in a polyphonic score

Expressive re-performance

Thesis (S.M.)--Massachusetts Institute of Technology, School of Architecture and Planning, Program in Media Arts and Sciences, 2011.Cataloged from PDF version of thesis.Includes bibliographical references (p. 167-171).Many music enthusiasts abandon music studies because they are frustrated by the amount of time and effort it takes to learn to play interesting songs. There are two major components to performance: the technical requirement of correctly playing the notes, and the emotional content conveyed through expressivity. While technical details like pitch and note order are largely set, expression, which is accomplished through timing, dynamics, vibrato, and timbre, is more personal. This thesis develops expressive re-performance, which entails the simplification of technical requirements of music-making to allow a user to experience music beyond his technical level, with particular focus on expression. Expressive re-performance aims to capture the fantasy and sound of a favorite recording by using audio extraction to split the original target solo and giving expressive control over that solo to a user. The re-performance experience starts with an electronic mimic of a traditional instrument with which the user steps-through a recording. Data generated from the users actions is parsed to determine note changes and expressive intent. Pitch is innate to the recording, allowing the user to concentrate on expressive gesture. Two pre-processing systems, analysis to discover note starts and extraction, are necessary. Extraction of the solo is done through user provided mimicry of the target combined with Probabalistic Latent Component Analysis with Dirichlet Hyperparameters. Audio elongation to match the users performance is performed using time-stretch. Instrument interfaces used were Akais Electronic Wind Controller (EWI), Fender's Squier Stratocaster Guitar and Controller, and a Wii-mote. Tests of the system and concept were performed using the EWI and Wii-mote for re-performance of two songs. User response indicated that while the interaction was fun, it did not succeed at enabling significant expression. Users expressed difficulty learning to use the EWI during the short test window and had insufficient interest in the offered songs. Both problems should be possible to overcome with further test time and system development. Users expressed interest in the concept of a real instrument mimic and found the audio extractions to be sufficient. Follow-on work to address issues discovered during the testing phase is needed to further validate the concept and explore means of developing expressive re-performance as a learning tool.by Laurel S. Pardue.S.M

Detección e identificación de señales sonoras en entornos asistivos.

150 p.El trabajo desarrollado en este documento de Tesis Doctoral tiene como principal objetivo el estudio y aplicabilidad de técnicas de reconocimiento de sonidos no relacionados con el habla, tales como timbres de puerta, grifos abiertos, despertadores, etc., que ayuden a mejorar la independencia y calidad de vida de las personas con discapacidad auditiva.En esta investigación se han desarrollado sistemas de reconocimiento capaces de trabajar en tiempo real utilizando micrófonos profesionales con una localización fija. Estos sistemas han sido diseñados tanto para avisar a las personas con problemas auditivos de sonidos de interés como para su uso en sistemas inteligentes que utilicen esta información para el reconocimiento de actividades de la vida diaria de la persona. No obstante, la principal contribución de esta tesis reside en la investigación de este tipo de sistemas en teléfonos móviles donde las prestaciones hardware están más limitadas y las condiciones de entrenamiento de los sonidos y las de validación o testeo varían. Se ha demostrado cómo optimizando los algoritmos de detección y clasificación, estos sistemas pueden ser funcionales en dispositivos móviles en tiempo real. El trabajo en este campo ha derivado en el desarrollo de una aplicación funcional para teléfonos móviles, capaz de funcionar en tiempo real y diseñada en base a pautas de accesibilidad para el apoyo de personas con discapacidad auditiva

Musical Instrument Classification Using Individual Partials

In a musical signals, the spectral and temporal contents of instruments often overlap. If the number of channels is at least the same as the number of instruments, it is possible to apply statistical tools to highlight the characteristics of each instrument, making their identification possible. However, in the underdetermined case, in which there are fewer channels than sources, the task becomes challenging. One possible way to solve this problem is to seek for regions in the time and/or frequency domains in which the content of a given instrument appears isolated. The strategy presented in this paper explores the spectral disjointness among instruments by identifying isolated partials, from which a number of features are extracted. The information contained in those features, in turn, is used to infer which instrument is more likely to have generated that partial. Hence, the only condition for the method to work is that at least one isolated partial exists for each instrument somewhere in the signal. If several isolated partials are available, the results are summarized into a single, more accurate classification. Experimental results using 25 instruments demonstrate the good discrimination capabilities of the method. © 2010 IEEE.191111122Agostini, G., Longari, M., Pollastri, E., Musical instrument timbres classification with spectral features (2003) EURASIP J. Appl. Signal Process., 2003, pp. 5-14Benetos, E., Kotti, M., Kotropoulos, C., Musical instrument classification using non-negative matrix factorization algorithms and subset feature selection (2006) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 221-224Chetry, N., Sandler, M., Linear predictive models for musical instrument identification (2009) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 173-176Eronen, A., Klapuri, A., Musical instrument recognition using cepstral coefficients and temporal features (2000) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 753-756Essid, S., Richard, G., David, B., Musical instrument recognition by pairwise classification strategies (2006) IEEE Trans. Audio, Speech, Lang. Process., 14 (4), pp. 1401-1412. , JulFanelli, A.M., Caponetti, L., Castellano, G., Buscicchio, C.A., Content-based recognition of musical instruments (2004) Proc. IEEE Int. Symp. Signal Process. Inf. Tech., pp. 361-364Ihara, M., Maeda, S.-I., Ishii, S., Instrument identification in monophonic music using spectral information (2007) Proc. IEEE Int. Symp. Signal Process. Inf. Tech., pp. 595-599Joder, C., Essid, S., Richard, G., Temporal integration for audio classification with application to musical instrument classification (2009) IEEE Trans. Audio, Speech, Lang. Process., 17 (1), pp. 174-186. , JanKrishna, A.G., Sreenivas, T.V., Music instrument recognition: From isolated notes to solo phrases (2004) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 265-268Martin, K.D., Kim, Y.E., Musical instrument identification: A pattern-recognition approach (1998) Proc. Meeting Acoust. Soc. Amer.Pruysers, C., Schnapp, J., Kaminskyj, I., Wavelet analysis in musical instrument sound classification (1998) Proc. Int. Symp. Signal Process. Applicat.Brown, J.C., Computer identification of musical instruments using pattern recognition with cepstral coefficients as features (1999) J. Acoust. Soc. Amer., 105, pp. 1933-1941Brown, J.C., Houix, O., McAdams, S., Feature dependence in the automatic identification of musical woodwind instruments (2001) J. Acoust. Soc. Amer., 109, pp. 1064-1072Fragoulis, D., Papaodysseus, C., Exarhos, M., Roussopoulos, G., Panagopoulos, T., Kamarotos, D., Automated classification of piano-guitar notes (2006) IEEE Trans. Audio, Speech, Lang. Process., 14 (3), pp. 1040-1050. , MayBurred, J.J., Robel, A., Sikora, T., Polyphonic musical instrument recognition based on a dynamic model of the spectral envelope (2009) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 173-176Jincahitra, P., Polyphonic instrument identification using independent subspace analysis (2004) Proc. IEEE Int. Conf. Multimedia Expo, pp. 1211-1214Kashino, K., Murase, H., A sound source identification system for ensemble music based on template adaptation and music stream extraction (1999) Speech Commun., 27, pp. 337-349Kitahara, T., Goto, M., Komatani, K., Ogata, T., Okuno, H.G., Instrument identification in polyphonic music: Feature weighting to minimize influence of sound overlaps (2007) EURASIP J. Appl. Signal Process., 2007, pp. 1-15Martins, L.G., Burred, J.J., Tzanetakis, G., Lagrange, M., Polyphonic instrument recognition using spectral clustering (2007) Proc. Int. Conf. Music Inf. RetrievalVincent, E., Rodet, X., Instrument identification in solo and ensemble music using independent subspace analysis (2004) Proc. Int. Conf. Music Inf. Retrieval, pp. 576-581Eggink, J., Brown, G.J., Application of missing feature theory to the recognition of musical instruments in polyphonic audio (2003) Proc. Int. Conf. Music Inf. Retrieval, pp. 1-7Eggink, J., Brown, G.J., Instrument recognition in accompanied sonatas and concertos (2004) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 217-220Leveau, P., Sodoyer, D., Daudet, L., Automatic instrument recognition in a polyphonic mixture using sparse representations (2007) Proc. Int. Conf. Music Inf. RetrievalEssid, S., Richard, G., David, B., Instrument recognition in polyphonic music based on automatic taxonomies (2006) IEEE Trans. Audio, Speech, Lang. Process., 14 (1), pp. 68-80. , JanSomerville, P., Uitdenbogerd, A.L., Multitimbral musical instrument classification (2008) Proc. Int. Symp. Comp. Science Applic., pp. 269-274Eggink, J., Brown, G.J., A missing feature approach to instrument identification in polyphonic music (2003) Proc. IEEE Int. Conf. Acoust., Speech, Signal Process., pp. 553-556Kostek, B., Musical instrument classification and duet analysis employing music information retrieval techniques (2004) Proc. IEEE, 92 (4), pp. 712-729. , AprThornburg, H., Leistikow, R.J., Berger, J., Melody extraction and musical onset detection via probabilistic models of framewise STFT peak data (2007) IEEE Trans. Audio, Speech, Lang. Process., 15 (4), pp. 1257-1272. , MayHu, G., Wang, D., Auditory segmentation based on onset and offset analysis (2007) IEEE Trans. Audio, Speech, Lang. Process., 15 (2), pp. 396-405. , FebZhou, R., Mattavelli, M., Zoia, G., Music onset detection based on resonator time frequency image (2008) IEEE Trans. Audio, Speech, Lang. Process., 16, pp. 1685-1695Barbedo, J.G.A., Lopes, A., Wolfe, P.J., Empirical methods to determine the number of sources in single-channel musical signals (2009) IEEE Trans. Audio, Speech, Lang. Process., 17 (8), pp. 1435-1444. , NovBarbedo, J.G.A., Lopes, A., Wolfe, P.J., High time-resolution estimation of multiple fundamental frequencies (2007) Proc. Int. Conf. Music Inf. Retrieval, pp. 399-403Rauhala, J., Lehtonen, H.-M., Valimaki, V., Fast automatic inharmonicity estimation algorithm (2007) J. Acoust. Soc. Amer., 121, pp. EL184-EL189Brown, J.C., Frequency ratios of spectral components of musical sounds (1996) J. Acoust. Soc. Amer., 99, pp. 1210-1218Oppenheim, A.V., Schafer, R.W., (1975) Digital Signal Processing, , Englewood Cliffs, NJ: Prentice-HallSmith III, J.O., Spectral Audio Signal Processing, , https://ccrma.stanford.edu/~jos/sasp, Online. AvailableEronen, A., Comparison of features for music instrument recognition (2001) Proc. IEEE Workshop Applicat. Signal Process Audio Acoust., pp. 19-22Every, M.R., Discriminating between pitched sources in music audio (2008) IEEE Trans. Audio, Speech, Lang. Process., 16 (2), pp. 267-277. , FebDeng, J.D., Simmermacher, C., Cranefield, S., A study on feature analysis for musical instrument classification (2008) IEEE Trans. Syst., Man. Cybern. B: Cybern., 38 (2), pp. 429-438. , AprGoto, M., Development of the RWC music database (2004) Proc. Int. Cong. Acoustics, pp. 553-556http://theremin.music.uiowa.edu, University of Iowa Musical Instrument Samples Database. Online. AvailableElliott, C., Attacks and releases as factors in instrument identification (1975) J. Res. Music Educat., 23, pp. 35-40Kendall, R.A., The role of acoustic signal partitions in listener categorization of musical phrases (1986) Music Percep., 4, pp. 185-214Martin, K.D., (1999) Sound-source Recognition: Atheory and Computational Model, , Ph. D. dissertation, Mass. Inst. of Technol., Cambridge, U. KSrinivasan, A., Sullivan, D., Fujinaga, I., Recognition of isolated instrument tones by conservatory students (2002) Proc. Int. Conf. Music Percept. Cogn., pp. 17-21Livshin, A., Rodet, X., The importance of cross database evaluation in sound classification (2003) Proc. Int. Conf. Music Inf. RetrievalEronen, A., (2001) Automatic Musical Instrument Recognition, , Ph. D. dissertation, Tampere Univ. of Technol., Tampere, FinlandEvery, M.R., (2006) Separation of Musical Sources and Structure From Single-channel Polyphonic Recordings, , Ph. D. dissertation, Univ. of York, York, U. KWu, M., Wang, D.L., Brown, G.J., A multi-pitch tracking algorithm for noisy speech (2003) IEEE Trans. Speech Audio Process., 11 (3), pp. 229-241. , Ma