Multimodal music information processing and retrieval: survey and future challenges

Towards improving the performance in various music information processing tasks, recent studies exploit different modalities able to capture diverse aspects of music. Such modalities include audio recordings, symbolic music scores, mid-level representations, motion, and gestural data, video recordings, editorial or cultural tags, lyrics and album cover arts. This paper critically reviews the various approaches adopted in Music Information Processing and Retrieval and highlights how multimodal algorithms can help Music Computing applications. First, we categorize the related literature based on the application they address. Subsequently, we analyze existing information fusion approaches, and we conclude with the set of challenges that Music Information Retrieval and Sound and Music Computing research communities should focus in the next years

The unbearable betweenness of being: Stories of identity and deviance between classical and jazz violin improvisation.

Abstract As a classically-trained violinist I experienced a creative and methodological tension when entering the field of jazz improvisation, relating to four different aspects: (1) The tertiary jazz environment required chord-chart literacy and refined aural skills, (2) The tertiary jazz environment emphasised rhythmic and chromatic language that was far removed from classical repertoire, (3) The generation of spontaneous musical ideas was difficult due in part to perfectionist ideals cultivated in classical training, and (4) Fingering challenges would arise when improvising through flat and sharp chords. My motivation for this project has been to resolve these tensions, and discover an idiolectal voice that synthesises aspects of both classical violin and jazz improvisation. I have constructed a Method that utilises tonal motifs, extracted from classical repertoire, to create solo violin etudes over the harmony from jazz standards. This Method is divided into three Levels, each of which aim to increase necessary skills for improvising. Level One deals with functional and technical concerns. These include chord-chart literacy, observing correct chord-scale relationships, and violin-specific fingering concerns when playing through disparate tonal centres. Level Two builds upon these skills, while introducing voice-leading and audiation skills. Levels Two and Three incorporate aesthetic concerns, where musical 'storytelling' -the thematic, individualistic development of ideas - is paramount. In Level Two, classical motifs are used as inspiration to develop consonant stories. Level Three engages with cross-modal abstraction to develop unconventional, and potentially dissonant (understood here as 'deviant') stories. Level Three transcends the technical and functional, concerned instead with phenomenological experience in improvisation and composition. This submission for the degree of Masters of Philosophy consists of a portfolio of original compositions and recorded works realised through practice-led research. The exegesis is divided into three parts. Part One presents research relevant to the situation of classically-trained improvisers, identifying challenges that classical musicians face when seeking to gain hybrid jazz skills. Part Two details various deliberate practice strategies devised to meet the primary challenges of jazz improvisation. Part Three discusses performed improvisations actualised through Levels Two and Three of the Method. Separate to the exegesis, a compendium is included, where notated original works and practice exercises are detailed. This dissertation provides insights into the creative processes that are often undefined in improvisation, processes that can facilitate an emergent and intentional musical identity. This research explores modalities and methods available to improvisers who wish to transform the tonal beauty of classical motifs into improvised musical works

Biomechanical Modelling of Musical Performance: A Case Study of the Guitar

Merged with duplicate record 10026.1/2517 on 07.20.2017 by CS (TIS)Computer-generated musical performances are often criticised for being unable to match the expressivity found in performances by humans. Much research has been conducted in the past two decades in order to create computer technology able to perform a given piece music as expressively as humans, largely without success. Two approaches have been often adopted to research into modelling expressive music performance on computers. The first focuses on sound; that is, on modelling patterns of deviations between a recorded human performance and the music score. The second focuses on modelling the cognitive processes involved in a musical performance. Both approaches are valid and can complement each other. In this thesis we propose a third complementary approach, focusing on the guitar, which concerns the physical manipulation of the instrument by the performer: a biomechanical approach. The essence of this thesis is a study on capturing, analyzing and modelling information about motor and biomechanical processes of guitar performance. The focus is on speed, precision, and force of a guitarist's left-hand. The overarching questions behind our study are: 1) Do unintentional actions originating from motor and biomechanical functions during musical performance contribute a material "human feel" to the performance? 2) Would it be possible determine and quantify such unintentional actions? 3) Would it be possible to model and embed such information in a computer system? The contributionst o knowledgep ursued in this thesis include: a) An unprecedented study of guitar mechanics, ergonomics, and playability; b) A detailed study of how the human body performs actions when playing the guitar; c) A methodologyt o formally record quantifiable data about such actionsin performance; d) An approach to model such information, and e) A demonstration of how the above knowledge can be embeddedin a system for music performance

EVERYTHING IS IMPORTANT

Composition for voice, string quartet and film. 41 minutes in duration

Automatic Transcription of Bass Guitar Tracks applied for Music Genre Classification and Sound Synthesis

Musiksignale bestehen in der Regel aus einer Überlagerung mehrerer Einzelinstrumente. Die meisten existierenden Algorithmen zur automatischen Transkription und Analyse von Musikaufnahmen im Forschungsfeld des Music Information Retrieval (MIR) versuchen, semantische Information direkt aus diesen gemischten Signalen zu extrahieren. In den letzten Jahren wurde häufig beobachtet, dass die Leistungsfähigkeit dieser Algorithmen durch die Signalüberlagerungen und den daraus resultierenden Informationsverlust generell limitiert ist. Ein möglicher Lösungsansatz besteht darin, mittels Verfahren der Quellentrennung die beteiligten Instrumente vor der Analyse klanglich zu isolieren. Die Leistungsfähigkeit dieser Algorithmen ist zum aktuellen Stand der Technik jedoch nicht immer ausreichend, um eine sehr gute Trennung der Einzelquellen zu ermöglichen. In dieser Arbeit werden daher ausschließlich isolierte Instrumentalaufnahmen untersucht, die klanglich nicht von anderen Instrumenten überlagert sind. Exemplarisch werden anhand der elektrischen Bassgitarre auf die Klangerzeugung dieses Instrumentes hin spezialisierte Analyse- und Klangsynthesealgorithmen entwickelt und evaluiert.Im ersten Teil der vorliegenden Arbeit wird ein Algorithmus vorgestellt, der eine automatische Transkription von Bassgitarrenaufnahmen durchführt. Dabei wird das Audiosignal durch verschiedene Klangereignisse beschrieben, welche den gespielten Noten auf dem Instrument entsprechen. Neben den üblichen Notenparametern Anfang, Dauer, Lautstärke und Tonhöhe werden dabei auch instrumentenspezifische Parameter wie die verwendeten Spieltechniken sowie die Saiten- und Bundlage auf dem Instrument automatisch extrahiert. Evaluationsexperimente anhand zweier neu erstellter Audiodatensätze belegen, dass der vorgestellte Transkriptionsalgorithmus auf einem Datensatz von realistischen Bassgitarrenaufnahmen eine höhere Erkennungsgenauigkeit erreichen kann als drei existierende Algorithmen aus dem Stand der Technik. Die Schätzung der instrumentenspezifischen Parameter kann insbesondere für isolierte Einzelnoten mit einer hohen Güte durchgeführt werden.Im zweiten Teil der Arbeit wird untersucht, wie aus einer Notendarstellung typischer sich wieder- holender Basslinien auf das Musikgenre geschlossen werden kann. Dabei werden Audiomerkmale extrahiert, welche verschiedene tonale, rhythmische, und strukturelle Eigenschaften von Basslinien quantitativ beschreiben. Mit Hilfe eines neu erstellten Datensatzes von 520 typischen Basslinien aus 13 verschiedenen Musikgenres wurden drei verschiedene Ansätze für die automatische Genreklassifikation verglichen. Dabei zeigte sich, dass mit Hilfe eines regelbasierten Klassifikationsverfahrens nur Anhand der Analyse der Basslinie eines Musikstückes bereits eine mittlere Erkennungsrate von 64,8 % erreicht werden konnte.Die Re-synthese der originalen Bassspuren basierend auf den extrahierten Notenparametern wird im dritten Teil der Arbeit untersucht. Dabei wird ein neuer Audiosynthesealgorithmus vorgestellt, der basierend auf dem Prinzip des Physical Modeling verschiedene Aspekte der für die Bassgitarre charakteristische Klangerzeugung wie Saitenanregung, Dämpfung, Kollision zwischen Saite und Bund sowie dem Tonabnehmerverhalten nachbildet. Weiterhin wird ein parametrischerAudiokodierungsansatz diskutiert, der es erlaubt, Bassgitarrenspuren nur anhand der ermittel- ten notenweisen Parameter zu übertragen um sie auf Dekoderseite wieder zu resynthetisieren. Die Ergebnisse mehrerer Hötest belegen, dass der vorgeschlagene Synthesealgorithmus eine Re- Synthese von Bassgitarrenaufnahmen mit einer besseren Klangqualität ermöglicht als die Übertragung der Audiodaten mit existierenden Audiokodierungsverfahren, die auf sehr geringe Bitraten ein gestellt sind.Music recordings most often consist of multiple instrument signals, which overlap in time and frequency. In the field of Music Information Retrieval (MIR), existing algorithms for the automatic transcription and analysis of music recordings aim to extract semantic information from mixed audio signals. In the last years, it was frequently observed that the algorithm performance is limited due to the signal interference and the resulting loss of information. One common approach to solve this problem is to first apply source separation algorithms to isolate the present musical instrument signals before analyzing them individually. The performance of source separation algorithms strongly depends on the number of instruments as well as on the amount of spectral overlap.In this thesis, isolated instrumental tracks are analyzed in order to circumvent the challenges of source separation. Instead, the focus is on the development of instrument-centered signal processing algorithms for music transcription, musical analysis, as well as sound synthesis. The electric bass guitar is chosen as an example instrument. Its sound production principles are closely investigated and considered in the algorithmic design.In the first part of this thesis, an automatic music transcription algorithm for electric bass guitar recordings will be presented. The audio signal is interpreted as a sequence of sound events, which are described by various parameters. In addition to the conventionally used score-level parameters note onset, duration, loudness, and pitch, instrument-specific parameters such as the applied instrument playing techniques and the geometric position on the instrument fretboard will be extracted. Different evaluation experiments confirmed that the proposed transcription algorithm outperformed three state-of-the-art bass transcription algorithms for the transcription of realistic bass guitar recordings. The estimation of the instrument-level parameters works with high accuracy, in particular for isolated note samples.In the second part of the thesis, it will be investigated, whether the sole analysis of the bassline of a music piece allows to automatically classify its music genre. Different score-based audio features will be proposed that allow to quantify tonal, rhythmic, and structural properties of basslines. Based on a novel data set of 520 bassline transcriptions from 13 different music genres, three approaches for music genre classification were compared. A rule-based classification system could achieve a mean class accuracy of 64.8 % by only taking features into account that were extracted from the bassline of a music piece.The re-synthesis of a bass guitar recordings using the previously extracted note parameters will be studied in the third part of this thesis. Based on the physical modeling of string instruments, a novel sound synthesis algorithm tailored to the electric bass guitar will be presented. The algorithm mimics different aspects of the instrument’s sound production mechanism such as string excitement, string damping, string-fret collision, and the influence of the electro-magnetic pickup. Furthermore, a parametric audio coding approach will be discussed that allows to encode and transmit bass guitar tracks with a significantly smaller bit rate than conventional audio coding algorithms do. The results of different listening tests confirmed that a higher perceptual quality can be achieved if the original bass guitar recordings are encoded and re-synthesized using the proposed parametric audio codec instead of being encoded using conventional audio codecs at very low bit rate settings

Multisensory learning in adaptive interactive systems

The main purpose of my work is to investigate multisensory perceptual learning and sensory integration in the design and development of adaptive user interfaces for educational purposes. To this aim, starting from renewed understanding from neuroscience and cognitive science on multisensory perceptual learning and sensory integration, I developed a theoretical computational model for designing multimodal learning technologies that take into account these results. Main theoretical foundations of my research are multisensory perceptual learning theories and the research on sensory processing and integration, embodied cognition theories, computational models of non-verbal and emotion communication in full-body movement, and human-computer interaction models. Finally, a computational model was applied in two case studies, based on two EU ICT-H2020 Projects, "weDRAW" and "TELMI", on which I worked during the PhD

Volume 60, Number 03 (March 1942)

Proud to Be a Go-Between Forward March with Music! Highlights in the Art of Teaching the Piano Practical Steps Toward Better Singing (interview with Emma Otero) Dedicated to _______ How I Became an Opera Conductor (interview with Edwin McArthur) Returning to Vocal Fundamentals Cultural Value of Magazines in America: The Report of an Exhaustic Scientific Survey Conducted by Purdue University and Directed by Dr. H.H. Remmers and Dr. W.A. Kerr Recognition for the Composer Dear Harp of My Country: Tom Moore the Irish Minstrel Have You a Song in Your Heart? Outstanding Achievements of Negro Composers Try It in Your Community Technic of the Month—Double Not Staccato (Czerny, Op. 335, No. 42) Ferdinando Carulli, 1770-1841https://digitalcommons.gardner-webb.edu/etude/1240/thumbnail.jp

Violin Augmentation Techniques for Learning Assistance

PhDLearning violin is a challenging task requiring execution of pitch tasks with the left hand using a strong aural feedback loop for correctly adjusting pitch, concurrent with the right hand moving a bow precisely with correct pressure across strings. Real-time technological assistance can help a student gain feedback and understanding helpful for learning and maintaining motivation. This thesis presents real-time low-cost low-latency violin augmentations that can be used to assist learning the violin along with other real-time performance tasks. To capture bow performance, we demonstrate a new means of bow tracking by measuring bow hair de ection from the bow hair being pressed against the string. Using near- eld optical sensors placed along the bow we are able to estimate bow position and pressure through linear regression from training samples. For left hand pitch tracking, we introduce low cost means for tracking nger position and illustrate the combination of sensed results with audio processing to achieve high accuracy low-latency pitch tracking. We subsequently verify our new tracking methods' e ectiveness and usefulness demonstrating low-latency note onset detection and control of real-time performance visuals. To help tackle the challenge of intonation, we used our pitch estimation to develop low latency pitch correction. Using expert performers, we veri ed that fully correcting pitch is not only disconcerting but breaks a violinist's learned pitch feedback loop resulting in worse asplayed performance. However, partial pitch correction, though also linked to worse as-played performance, did not lead to a signi cantly negative experience con rming its potential for use to temporarily reduce barriers to success. Subsequently, in a study with beginners, we veri ed that when the pitch feedback loop is underdeveloped, automatic pitch correction did not signi cantly hinder performance, but o ered an enjoyable low-pitch error experience and that providing an automatic target guide pitch was helpful in correcting performed pitch error