A Human-Computer Duet System for Music Performance

Virtual musicians have become a remarkable phenomenon in the contemporary multimedia arts. However, most of the virtual musicians nowadays have not been endowed with abilities to create their own behaviors, or to perform music with human musicians. In this paper, we firstly create a virtual violinist, who can collaborate with a human pianist to perform chamber music automatically without any intervention. The system incorporates the techniques from various fields, including real-time music tracking, pose estimation, and body movement generation. In our system, the virtual musician's behavior is generated based on the given music audio alone, and such a system results in a low-cost, efficient and scalable way to produce human and virtual musicians' co-performance. The proposed system has been validated in public concerts. Objective quality assessment approaches and possible ways to systematically improve the system are also discussed

Alignment of the recording of a violin performance with the corresponding musical score

Català: En aquest projecte es pretén estudiar una part de la conducta dels músics, com és la variació del tempo al llarg de la cançó. Per fer-ho, s'ha d'obtenir la durada de les notes durant la interpretació de les peces. Les notes tenen una durada determinada a la partitura, però aquesta varia substancialment respecte la durada de les notes tocades pels músics, ja que aquests toquen sense l'ajut de metrònom. És per això que existeix una variació en el tempo de la cançó respecte el tempo teòric. Obtenint la durada real de les notes, es pot calcular el tempo seguit pel músic. En el nostre cas els músics estudiats són violinistes. El procediment per analitzar els temes musicals enregistrats consisteix en la gravació del so, i en la utilització d'uns sensors per obtenir la posició, el moviment, la força i la velocitat de l'arc durant la peça. Amb tots aquests paràmetres és possible aconseguir un model definit de la interpretació del músic. Per calcular el tempo s?ha implementat un algorisme, anomenat detector de polsos o beat detector, que detecta el tempo seguit pel violinista durant la interpretació. Posteriorment, s?ha implementat un altre algorisme, anomenat alineador o aligner que, aprofitant els resultats de la variació del tempo, crea una partitura amb les duracions reals de cada nota. Actualment, els sistemes existents que detecten el tempo de les cançons utilitzen només les dades de la gravació del so. Per tal de millorar l'eficiència d'aquests algorismes, proposem un nou sistema que, a part d'utilitzar el so, incorpora les dades provinents del moviment de l'arc. Finalment, per tal de millorar els resultats obtinguts, s'ha proposat un sistema que detecta els polsos de les cançons utilitzant les dades provinents d'un duet, trio o quartet com a senyals d'entrada. El fet que hi hagi més d'una veu en una cançó implica que hi hagi una relació implícita entre els tempos que segueixen els violinistes, que és la que utilitza l'algorisme proposat.Castellano: En este proyecto se pretende estudiar una parte de la conducta de los músicos, como es la variación del tempo a lo largo de la canción. Para hacerlo, se obtiene la duración de las notas durante la interpretación de las piezas. Las notas tienen una duración determinada en la partitura, pero ésta varia sustancialmente respecto la duración de las notas tocadas por los músicos, ya que éstos tocan sin la ayuda de metrónomo. Es por eso que existe una variación en el tempo de la canción respecto al tempo teórico. Obteniendo la duración real de las notas, se puede calcular el tempo seguido por el músico. En nuestro caso los músicos estudiados son violinistas. El procedimiento para analizar los temas musicales grabados consiste en la grabación del sonido y en la utilización de unos sensores para obtener la posición, el movimiento, la fuerza y la velocidad del arco durante la pieza. Con todos estos parámetros es posible conseguir un modelo definido de la interpretación del músico. Actualmente, los sistemas existentes que detectan el tempo de las canciones utilizan sólo los datos de la grabación del sonido. Con tal de mejorar la eficiencia de estos algoritmos, proponemos un nuevo sistema que, a parte de utilizar el sonido, incorpora los datos provenientes del movimiento del arco. Finalmente, con tal de mejorar los resultados obtenidos, se propone un sistema que detecta los pulsos de las canciones utilizando los datos provenientes de un dueto, trío o cuarteto como señales de entrada. El hecho de que haya más de una voz en una canción implica que haya una relación implícita entres los tempos que siguen los violinistas, que es la que utiliza el algoritmo propuesto.English: This work aims to study some of the behavior of the musicians regarding the change of the tempo throughout the song. In order to do it, the length of the notes during the performance of the pieces has to be computed. Although the notes have a fixed length marked in the score, it varies substantially with the duration of the notes of the song. Because they play without the help of a metronome, this is why there is a variation of the tempo of the songs compared with the theoretical one. The actual tempo followed by the musicians can be computed by calculating the actual duration of the notes. In our case, the musicians studied are violinists. The procedure to analyze the recorded songs consists of using some sensors to obtain the position, the velocity, the movement and the force of the bow used in the performance. With all these parameters it is possible to define an accurate model of the musical performance. Currently, the existing systems that detect the tempo of the songs only use the recording data as an input. In order to improve the efficiency of these algorithms, we propose a new system that not only uses the recording, but also the data of the bow displacement. Finally, in order to improve the obtained results, we propose another system that detects the beats of the songs using the data of a duet, trio or quartet as inputs. In these cases, there is more than one violin playing at the same time, and there is an implicit relation between the tempos followed by the violinists. The proposed algorithm takes advantage of this relation to obtain the actual tempo

Estimation of Guitar Fingering and Plucking Controls based on Multimodal Analysis of Motion, Audio and Musical Score

This work presents a method for the extraction of instrumental controls during guitar performances. The method is based on the analysis of multimodal data consisting of a combination of motion capture, audio analysis and musical score. High speed video cameras based on marker identification are used to track the position of finger bones and articulations and audio is recorded with a transducer measuring vibration on the guitar body. The extracted parameters are divided into left hand controls, i.e. fingering (which string and fret is pressed with a left hand finger) and right hand controls, i.e. the plucked string, the plucking finger and the characteristics of the pluck (position, velocity and angles with respect to the string). Controls are estimated based on probability functions of low level features, namely, the plucking instants (i.e. note onsets), the pitch and the distances of the fingers (both hands) to strings and frets. Note onsets are detected via audio analysis, the pitch is extracted from the score and distances are computed from 3D Euclidean Geometry. Results show that by combination of multimodal information, it is possible to estimate such a comprehensive set of control features, with special high performance for the fingering and plucked string estimation. Regarding the plucking finger and the pluck characteristics, their accuracy gets lower but improvements are foreseen including a hand model and the use of high-speed cameras for calibration and evaluation.A. Perez-Carrillo was supported by a Beatriu de Pinos grant 2010 BP-A 00209 by the Catalan Research Agency (AGAUR) and J. Ll. Arcos was supported by ICT -2011-8-318770 and 2009-SGR-1434 projectsPeer reviewe

Measuring Expressive Music Performances: a Performance Science Model using Symbolic Approximation

Music Performance Science (MPS), sometimes termed systematic musicology in Northern Europe, is concerned with designing, testing and applying quantitative measurements to music performances. It has applications in art musics, jazz and other genres. It is least concerned with aesthetic judgements or with ontological considerations of artworks that stand alone from their instantiations in performances. Musicians deliver expressive performances by manipulating multiple, simultaneous variables including, but not limited to: tempo, acceleration and deceleration, dynamics, rates of change of dynamic levels, intonation and articulation. There are significant complexities when handling multivariate music datasets of significant scale. A critical issue in analyzing any types of large datasets is the likelihood of detecting meaningless relationships the more dimensions are included. One possible choice is to create algorithms that address both volume and complexity. Another, and the approach chosen here, is to apply techniques that reduce both the dimensionality and numerosity of the music datasets while assuring the statistical significance of results. This dissertation describes a flexible computational model, based on symbolic approximation of timeseries, that can extract time-related characteristics of music performances to generate performance fingerprints (dissimilarities from an ‘average performance’) to be used for comparative purposes. The model is applied to recordings of Arnold Schoenberg’s Phantasy for Violin with Piano Accompaniment, Opus 47 (1949), having initially been validated on Chopin Mazurkas.1 The results are subsequently used to test hypotheses about evolution in performance styles of the Phantasy since its composition. It is hoped that further research will examine other works and types of music in order to improve this model and make it useful to other music researchers. In addition to its benefits for performance analysis, it is suggested that the model has clear applications at least in music fraud detection, Music Information Retrieval (MIR) and in pedagogical applications for music education

A Weighted Individual Performance-Based Assessment for Middle School Orchestral Strings: Establishing Validity and Reliability

The study established the validity and reliability of a weighted individual performance-based assessment tool within the utility scope of middle school orchestral strings. The following research questions guided this study: 1. What specific string-playing behaviors and corresponding criteria validate a weighted individual performance-based assessment tool for middle school orchestral strings? 2. What are the psychometric properties of the weighted individual performance-based assessment tool in authentic situations? For Research Question 1, the expert panel and I were able to 100% mutually agree on 10 string-playing behaviors: tempo, rhythm, tone, pitch, intonation, technique, bowing, dynamics, phrasing, and posture that created the DISAT. Being interdependent, these string-playing behaviors are relevant because they encompass every necessary facet of orchestral string performance (Zdzinski & Barnes, 2002). According to Zdzinski and Barnes (2002), an orchestral string performance assessment must evaluate each facet of a participant’s playing ability to rate the overall musicianship. Bergee and Rossin (2019) stated in their research that it is important to have various aspects of a performance utilized in a musical assessment. The DISAT obtained reliability of 0.872 by having enough variance between raters in the authentic situation. Linacre (2015) stated that reliability greater than 0.8 is acceptable to v distinguish separation between raters. Combined with the expert panel\u27s 100% mutual agreement on content validity, this proved the DISAT to be a valid and reliable assessment tool for individual performance-based orchestral strings assessment (AERA, APA, & NCME, 2014). The DISAT can be utilized by districts and middle school orchestral string music teachers in North Carolina. Being a consistent, objective tool, the DISAT can standardize our approach to middle school orchestral string music education assessment (AERA, APA, & NCME, 2014). The data collected by the DISAT could easily track the musical progression of students while giving opportunities for constructive, purposeful feedback

On the analysis of musical performance by computer

Existing automatic methods of analysing musical performance can generally be described as music-oriented DSP analysis. However, this merely identifies attributes, or artefacts which can be found within the performance. This information, though invaluable, is not an analysis of the performance process. The process of performance first involves an analysis of the score (whether from a printed sheet or from memory), and through this analysis, the performer decides how to perform the piece. Thus, an analysis of the performance process requires an analysis of the performance attributes and artefacts in the context of the musical score. With this type analysis it is possible to ask profound questions such as “why or when does a performer use this technique”. The work presented in this thesis provides the tools which are required to investigate these performance issues. A new computer representation, Performance Markup Language (PML) is presented which combines the domains of the musical score, performance information and analytical structures. This representation provides the framework with which information within these domains can be cross-referenced internally, and the markup of information in external files. Most importantly, the rep resentation defines the relationship between performance events and the corresponding objects within the score, thus facilitating analysis of performance information in the context of the score and analyses of the score. To evaluate the correspondences between performance notes and notes within the score, the performance must be analysed using a score-performance matching algorithm. A new score-performance matching algorithm is presented in this document which is based on Dynamic Programming. In score-performance matching there are situations where dynamic programming alone is not sufficient to accurately identify correspondences. The algorithm presented here makes use of analyses of both the score and the performance to overcome the inherent shortcomings of the DP method and to improve the accuracy and robustness of DP matching in the presence of performance errors and expressive timing. Together with the musical score and performance markup, the correspondences identified by the matching algorithm provide the minimum information required to investigate musical performance, and forms the foundation of a PML representation. The Microtonalism project investigated the issues surrounding the performance of microtonal music on conventional (i.e. non microtonal specific) instruments, namely voice. This included the automatic analysis of vocal performances to extract information regarding pitch accuracy. This was possible using tools developed using the performance representation and the matching algorithm

Automatic Transcription of Bass Guitar Tracks applied for Music Genre Classification and Sound Synthesis

Musiksignale bestehen in der Regel aus einer Überlagerung mehrerer Einzelinstrumente. Die meisten existierenden Algorithmen zur automatischen Transkription und Analyse von Musikaufnahmen im Forschungsfeld des Music Information Retrieval (MIR) versuchen, semantische Information direkt aus diesen gemischten Signalen zu extrahieren. In den letzten Jahren wurde häufig beobachtet, dass die Leistungsfähigkeit dieser Algorithmen durch die Signalüberlagerungen und den daraus resultierenden Informationsverlust generell limitiert ist. Ein möglicher Lösungsansatz besteht darin, mittels Verfahren der Quellentrennung die beteiligten Instrumente vor der Analyse klanglich zu isolieren. Die Leistungsfähigkeit dieser Algorithmen ist zum aktuellen Stand der Technik jedoch nicht immer ausreichend, um eine sehr gute Trennung der Einzelquellen zu ermöglichen. In dieser Arbeit werden daher ausschließlich isolierte Instrumentalaufnahmen untersucht, die klanglich nicht von anderen Instrumenten überlagert sind. Exemplarisch werden anhand der elektrischen Bassgitarre auf die Klangerzeugung dieses Instrumentes hin spezialisierte Analyse- und Klangsynthesealgorithmen entwickelt und evaluiert.Im ersten Teil der vorliegenden Arbeit wird ein Algorithmus vorgestellt, der eine automatische Transkription von Bassgitarrenaufnahmen durchführt. Dabei wird das Audiosignal durch verschiedene Klangereignisse beschrieben, welche den gespielten Noten auf dem Instrument entsprechen. Neben den üblichen Notenparametern Anfang, Dauer, Lautstärke und Tonhöhe werden dabei auch instrumentenspezifische Parameter wie die verwendeten Spieltechniken sowie die Saiten- und Bundlage auf dem Instrument automatisch extrahiert. Evaluationsexperimente anhand zweier neu erstellter Audiodatensätze belegen, dass der vorgestellte Transkriptionsalgorithmus auf einem Datensatz von realistischen Bassgitarrenaufnahmen eine höhere Erkennungsgenauigkeit erreichen kann als drei existierende Algorithmen aus dem Stand der Technik. Die Schätzung der instrumentenspezifischen Parameter kann insbesondere für isolierte Einzelnoten mit einer hohen Güte durchgeführt werden.Im zweiten Teil der Arbeit wird untersucht, wie aus einer Notendarstellung typischer sich wieder- holender Basslinien auf das Musikgenre geschlossen werden kann. Dabei werden Audiomerkmale extrahiert, welche verschiedene tonale, rhythmische, und strukturelle Eigenschaften von Basslinien quantitativ beschreiben. Mit Hilfe eines neu erstellten Datensatzes von 520 typischen Basslinien aus 13 verschiedenen Musikgenres wurden drei verschiedene Ansätze für die automatische Genreklassifikation verglichen. Dabei zeigte sich, dass mit Hilfe eines regelbasierten Klassifikationsverfahrens nur Anhand der Analyse der Basslinie eines Musikstückes bereits eine mittlere Erkennungsrate von 64,8 % erreicht werden konnte.Die Re-synthese der originalen Bassspuren basierend auf den extrahierten Notenparametern wird im dritten Teil der Arbeit untersucht. Dabei wird ein neuer Audiosynthesealgorithmus vorgestellt, der basierend auf dem Prinzip des Physical Modeling verschiedene Aspekte der für die Bassgitarre charakteristische Klangerzeugung wie Saitenanregung, Dämpfung, Kollision zwischen Saite und Bund sowie dem Tonabnehmerverhalten nachbildet. Weiterhin wird ein parametrischerAudiokodierungsansatz diskutiert, der es erlaubt, Bassgitarrenspuren nur anhand der ermittel- ten notenweisen Parameter zu übertragen um sie auf Dekoderseite wieder zu resynthetisieren. Die Ergebnisse mehrerer Hötest belegen, dass der vorgeschlagene Synthesealgorithmus eine Re- Synthese von Bassgitarrenaufnahmen mit einer besseren Klangqualität ermöglicht als die Übertragung der Audiodaten mit existierenden Audiokodierungsverfahren, die auf sehr geringe Bitraten ein gestellt sind.Music recordings most often consist of multiple instrument signals, which overlap in time and frequency. In the field of Music Information Retrieval (MIR), existing algorithms for the automatic transcription and analysis of music recordings aim to extract semantic information from mixed audio signals. In the last years, it was frequently observed that the algorithm performance is limited due to the signal interference and the resulting loss of information. One common approach to solve this problem is to first apply source separation algorithms to isolate the present musical instrument signals before analyzing them individually. The performance of source separation algorithms strongly depends on the number of instruments as well as on the amount of spectral overlap.In this thesis, isolated instrumental tracks are analyzed in order to circumvent the challenges of source separation. Instead, the focus is on the development of instrument-centered signal processing algorithms for music transcription, musical analysis, as well as sound synthesis. The electric bass guitar is chosen as an example instrument. Its sound production principles are closely investigated and considered in the algorithmic design.In the first part of this thesis, an automatic music transcription algorithm for electric bass guitar recordings will be presented. The audio signal is interpreted as a sequence of sound events, which are described by various parameters. In addition to the conventionally used score-level parameters note onset, duration, loudness, and pitch, instrument-specific parameters such as the applied instrument playing techniques and the geometric position on the instrument fretboard will be extracted. Different evaluation experiments confirmed that the proposed transcription algorithm outperformed three state-of-the-art bass transcription algorithms for the transcription of realistic bass guitar recordings. The estimation of the instrument-level parameters works with high accuracy, in particular for isolated note samples.In the second part of the thesis, it will be investigated, whether the sole analysis of the bassline of a music piece allows to automatically classify its music genre. Different score-based audio features will be proposed that allow to quantify tonal, rhythmic, and structural properties of basslines. Based on a novel data set of 520 bassline transcriptions from 13 different music genres, three approaches for music genre classification were compared. A rule-based classification system could achieve a mean class accuracy of 64.8 % by only taking features into account that were extracted from the bassline of a music piece.The re-synthesis of a bass guitar recordings using the previously extracted note parameters will be studied in the third part of this thesis. Based on the physical modeling of string instruments, a novel sound synthesis algorithm tailored to the electric bass guitar will be presented. The algorithm mimics different aspects of the instrument’s sound production mechanism such as string excitement, string damping, string-fret collision, and the influence of the electro-magnetic pickup. Furthermore, a parametric audio coding approach will be discussed that allows to encode and transmit bass guitar tracks with a significantly smaller bit rate than conventional audio coding algorithms do. The results of different listening tests confirmed that a higher perceptual quality can be achieved if the original bass guitar recordings are encoded and re-synthesized using the proposed parametric audio codec instead of being encoded using conventional audio codecs at very low bit rate settings