M.I. Martin-Erdozain, I. Barbancho, A. Rosa-Pujazon, A.M. Barbancho, "Correction system for polyphonic piano recordings", XXVIII Simposium Nacional de la Unión Científica Internacional de Radio, Santiago de Compostela, España, 2013n this paper, a support tool for piano rehearsal
is presented. The system analyses a given piano polyphonic
recording to find the times, pitch and duration of the notes
and figures played, taking into account the possibility of playing
more than one note simultaneously as well as covering the whole
piano frequency range. In order to do so, the system uses
an onset detection algorithm to segment the input signal into
partitions which are then analysed in the time and frequency
domains. Then, the system correlates the data extracted from
the partitions with the score of the original piece, identifying
the positions and type of the mistakes performed by the user,
and providing her/him with the corresponding feedback. The
experiments conducted showed that the application is capable
of analysing a given recording and indicate the musician the
mistakes made.This work has been funded by the Ministerio de Economia y
Competitividad of the Spanish Government under Project No.
TIN2010-21089-C03-02 and by the Ministerio de Industria,
Turismo y Comercio under Project No. TSI-090100-2011-25

Barbancho-Perez, Ana Maria

Barbancho-Perez, Isabel

Martin Erdozain, Maria Isabel

Rosa-Pujazon, Alejandro

English

Barbancho-Pérez, Isabel

Barbancho-Pérez, Ana María

Repositorio Institucional Universidad de Málaga

CORRECTION SYSTEM FOR POLYPHONICPIANO RECORDINGSM. Isabel Martı´n Erdozain(1), Isabel Barbancho(1), Alejandro Rosa-Pujazon(1), Ana Marı´a Barbancho(1)marisamartin87@gmail.com, ibp@ic.uma.es, alejandror@uma.es, abp@ic.uma.es(1)Department of Ingenierı´a de Comunicaciones, Universidad de Ma´laga. Campus de Teatinos s/n, 29071 Ma´laga, SpainAbstract— In this paper, a support tool for piano rehearsalis presented. The system analyses a given piano polyphonicrecording to find the times, pitch and duration of the notesand figures played, taking into account the possibility of playingmore than one note simultaneously as well as covering the wholepiano frequency range. In order to do so, the system usesan onset detection algorithm to segment the input signal intopartitions which are then analysed in the time and frequencydomains. Then, the system correlates the data extracted fromthe partitions with the score of the original piece, identifyingthe positions and type of the mistakes performed by the user,and providing her/him with the corresponding feedback. Theexperiments conducted showed that the application is capableof analysing a given recording and indicate the musician themistakes made.I. INTRODUCTIONThe advances in information and communication technolo-gies in the recent years have spread massively to most aspectsof our everyday-life, including also the educational field, asthe use of Web 2.0 applications and emerging technologieshas been proven to provide useful support tools in the school,helping students in their learning process and empoweringtheir creativity [1], [2], [3], [4].However, when it comes to the field of music studies,this array of tools might prove to be partially lacking. Theuse of web resources and applications can constitute a wayto increase students’ motivation towards learning, and in theparticular case of music learning, it can be quite helpful in theearly stages offer better accessibility to the abstract conceptsof music theory. Learning music, however, relies heavily onpractising, and therefore it is usually necessary to have a tutoror expert to show the student which aspects they need toimprove, which mistakes they commonly perform, etc. In thissense, there is a need for a more specialised guide in musiclearning, which the conventional use of currently availableapplications fail to provide. Thus, to satisfy such needs, itis necessary to use more specialized tools and applicationsto provide a more specific interface for an adequate learningexperience.In this paper, we present a system that addresses this need,serving as a support tool for piano students, and allowing themto correct their mistakes when practising without the needof having another musician acting as an external reviewer.Concretely, the system allows for the analysis of a musicalpiano polyphonic recording to assess the correctness of theperformance. The system is capable of segmenting and iden-tifying the notes and figures played, and compares them withthe score of the piece, finding where the practician has madeany potential mistakes, as well as informing the student of thetypes of mistakes performed.The next section will present the technical details of the al-gorithms implemented to analyse a piano recording, includingthe onset detector and the procedures followed to identify notelength, pitch and time. The following section will cover howthe piece analysed is corrected according to the data extractedfrom the original score. Next, the results of the tests performedwith the system will be presented, and finally the article willend with a presentation of the conclusions extracted from thiswork.II. ANALYSIS OF THE RECORDED MUSICAL SIGNALIn order to properly process a given musical piece, a time-frequency analysis is performed to find the notes played ateach time. The system loads a piano recording from a wavfile, with a sampling rate of 44100 Hz and a beats-per-minutevalue specified by the user. Then, the system divides the signalinto temporal slots or ”partitions”, each of these correspondingto the time at which a given note is being played, andanalyses the partition to find the pitch and duration of thenote(s) played. The system designed takes into account thepossibility of more than one note being played simultaneouslyand covers the whole frequency range of the piano. In thenext following subsections, the most relevant features of thesystem’s processing stage are presented.A. Temporal segmentationAn onset detector [5], [6], [7] has been implemented inorder to divide the musical signal into the aforementionedpartitions, each onset corresponding to the time instant inwhich a note ”attacks” or ”appears” in the signal, i.e. whenan energy peak corresponding to that note is introduced. Thesegmentation process consists of two major steps: the onsetdetection itself (to find the attack time of the note) and thedelimitation of each attack slot (to find when the note ”ends”).Before performing this segmentation, the signal is normalizedto have an amplitude between 0 and 1.General onsets and onsets masked by other onsets arelocated, and then, of all detected onsets, actual onsets aredetermined.Fig. 1. Onset detection standard processEi represents the energy for the i-th window, while the different µj valuesrepresent the thresholds used to find whether an onset is present or notIn the onset detection step, a sliding window procedure akinto the one described in [8] is used to detect energy peaks. Foreach sliding window, the energy Ei is calculated as follows:Ei =xi+L−1∑j=xi(y(j))2, (1)where xi is the index for the initial sample of window i, L isthe length of the window in samples and y(j) represents thejth-sample of the piano piece.We set a threshold value of 0.7 so that only those windowswhose Ei exceeds this threshold can potentially contain anonset. According to the ADSR (attack-decay-sustain-release)model [8], it may be possible that part of one note’s energyoverlaps with the next one, or that a increase of energy isfound because of the sustain-release of a previously detectednote (false onset). To account for this, the energy of eachwindow is compared with the energy had for the previous andnext ones, as per the graph presented in Fig. 1, where µ0 isthe aforementioned 0.7 threshold value, and the parameters µ1to µ3 define conditions to address the previously commentedissue.The different parameters were set to L=3000 samples,µ1=6.05, µ2=1.9 and µ3=3 to find the onsets in the piececonsidered. It may be possible though that some onsets weremasked by higher energy neighbouring notes for the windowsize considered. To prevent this issue, a second search formasked onsets is subsequently performed, this time with L =2000 samples and µ1=5.5.The piano signal is windowed without overlapping if noattacks are detected. However, if an attack is found in the ith-window, the next window is set to start 20% windows samplesbefore the location of the maximum amplitude sample foundin the ith-window.Finally, one single note might generate more than one onsetpeak in a short time period, but the note itself only has onereal onset time. To detect and erase subsequent false onsets,a minimum separation distance is defined according to theshortest figure’s duration considered in the piece used. Inparticular, following the ADSR model, the decay and releasetime is assumed to last 2/3 times the duration of the figure.The minimum separation is then defined as two thirds of theshortest figure duration in the score. Onsets that are separatedFig. 2. Input signal (up) and its corresponding detected onsets (down)less that this minimum are combined into a single onsetattack, storing only the onset that has the biggest energy valueassociated.After the different onsets have been found, the samples ofthe musical piece are divided into partitions according to theseonsets. Concretely, following the ADSR model, there is a briefattack time before reaching the maximum amplitude (onset),which is modelled by the system by considering that a givenpartition starts 1000 samples before the onset time itself. Inorder to prevent overlapping between two consecutive notesnear in time, the partition is set to finish 3200 samples beforethe next onset, as this number guarantees isolation betweenone note and the next one for the worst case scenario inthe range of frequencies of the piano (which was found at72.4 milliseconds, 3195 samples roughly at a sampling rateof 44100 Hz). An example of the output of the onset detectorimplemented can be seen in Fig. 2B. Temporal analysis: finding the duration of the notesplayedAfter finding the onsets and dividing the signal into parti-tions, the system has an effective segmentation of the notesplayed at each time. The next step needed is to find whichis the duration of each of the notes found. This is easilyachieved by simply dividing the number of samples in eachpartition by the sampling rate in the recording (by default,44100 Hz). However, this measure of duration is dependent onthe tempo at which the piece is being played. Thus, in order toobjectively determine the duration of the notes independentlyof the velocity in the performance, this measure of duration isnormalized by dividing it by the black figure’s duration (whichis actually the beats-per-minute value specified by the user).As each partition normalized duration is calculated, the figu-re played in that partition is then classified as one of possiblefigures in the score (whole, half, black, half-time, quarter-time,etc.), assigning it the type of figure whose normalized durationis closest to. The system also takes into account the possibilityof having dotted notes in the classification process.C. Frequency analysisThe time of the notes as well as their duration has alreadybeen found using the previous modules. The last step tofully characterize the score of the piece recorded is to findthe pitch of the notes actually played. This is performed bytransforming each partition into the frequency domain by usinga DFT (Discrete Fourier Transform). The resulting spectrumis normalized so that the frequency axis is scaled into a MIDInumbers axis.From this spectrum, a peak detection process is performedto find the pattern of peaks associated with each partition (thatis, which fundamental frequencies and partials are present inthe partition), normalizing the amplitudes of the peaks foundto the value of 1. This pattern of peaks is then correlated withthe pattern of peaks that correspond to each of the notes thatshould be played in that instant according to the score of thepiece assessed. If a given note was correctly played, then itspattern of peaks should be included in the pattern found forthe corresponding partition.III. ASSESSMENT OF CORRECTIONA. Parameters of a piano pieceThe previous section presented the most relevant aspect ofthe analysing blocks of the system that allow as to identifythe notes being played, their position and duration. In order tocheck the correctness of the piece recorded, the system alsorequires to have additional parameters specified regarding theoriginal score of the piece. Concretely, the system extractsthe required information regarding the time signature, musicalfigures and notes from an auxiliary data base.In this regard, the system stores information correspondingto the times of each note as well as their normalized duration.If two or more notes are played at the same time, the systemwill only consider the figure with shorter duration.The system identifies which notes are being played at eachinstant according to the peaks found in the MIDI spectrum.Thus, for each note or group of notes present in the score,its corresponding pattern of fundamental and partial spectralpeaks is stored as a reference. To account for potential varia-tions in the frequency values of the peaks due to inharmonicity[9], the reference pattern stored is not actually a pattern ofpeaks, but rather a pattern of narrow filters of width 1 (in theMIDI scale) centered at the frequencies were the fundamentaland partial peaks should lie in theory.B. Evaluation and correctionThe system uses the data stored as a reference and theinformation extracted from the analysis stage to judge thecorrectness of the piece recorded. Two correction processeshave been implemented, depending on whether the numberof note times identified is coincident with the number ofpartitions found.1. Normal correction: If the number of partitions and notetimes is the same, the system evaluates if the duration of thenotes in the performance is correct by simply comparing theduration estimated in each partition with the expected one inthe reference.To verify if the notes have been played at their correct fre-quencies, the pattern of peaks of each partitions is filtered bythe corresponding reference pattern, and the resulting spectrumis summed along all frequencies. If both patterns match, theexpected sum will be zero, otherwise the notes played werenot correct (there are peaks in the partition spectrum that donot have their match in the reference pattern)2. Special correction: The special correction is performedwhen the number of partitions is different from the number ofnote times. Given a piano performance, an error in the musicalfigure played is more probable than an error in the musical noteplayed. Therefore, the latter will be given more importancethan the former, as it is associated with a more meaningfulmistake. When assessing each partition’s correctness, there arethree possible cases:The notes played are the expected ones: in this case, thesystem proceeds as per the normal case.The notes are different from the expected ones and thereare more notes in the recording than in the score: thishappens because new notes have been added or the samenote has been played more than once. The notes playedin the actual partition are then compared with the onesexpected for the next one. If there is a coincidence, thenit is assumed that the user played a new non-existent notein the current partition. Otherwise, it is assumed that theuser simply played the note wrong.The notes are different from the expected ones and thereare less notes in the recording than in the score: this willhappen if the user skipped some notes when performing.Again, the next expected note in the score is comparedwith the note had in the actual partition. If they arethe same, it is assumed that the user skipped one note,otherwise it is again assumed that the user played thenote wrong.IV. RESULTSA. Onset detectionIt is crucial to corroborate that the onset detection algorithmproposed is indeed capable of adequately segmenting thesignal and identifying the partitions which the score consistsof. Thus, we conducted a set of tests in which a set of tenquavers or quarter-notes were played at different velocities,ranging from 40 to 230 beats-per-minute. For each tempoconsidered, three different indicators are used to assess thequality of the onset detector: the rate of detected notes over thetotal number of notes played (denoted by N ), the rate of falsenotes detected (false negatives, FN ), and a score indicatordefined as follows:Score =NN + FP + FN× 100, (2)where FP represents the false positives rate (non-existentonsets detected). The results yielded are summarized in Fig. 3.The onset detector used shows to be very effective at findingthe notes being played as long as the tempo of the piece isnot higher than 180 bpm. For faster pieces, the quality of thedetection worsens gradually.Fig. 3. Results for the evaluation of the onset detectorN = the rate of detected notes over the total number of notes, FN = therate of false negatives), Score = NN+FP+FN× 100 with FP being therate of false positivesFig. 4. Results for the evaluation of the onset detector:each bar color is associated to one type of error, i.e. dark blue - no error,orange - wrong duration of note, red - wrong note, cyan - skipped noteB. Correctness evaluationIn order to assess the viability of the global system as asupport tool for aided rehearsal, we conducted an experimentin which the song ’menuet 114’ was played and recorded at 21different speeds. The system presents the user a colored imagethat indicates him/her the types of errors that have been foundin his/her performance. An example can be found in the figureFig. 4. It was found that users tend to perform more errorswhen the tempo of the piece is higher, which is something tobe expected; also, as can be noticed, the vast majorities of themistakes made in the performance come from not keeping theduration of the figures as indicated in the score.From the tests performed, it was found that the systemdetected correctly the notes played most of the time (as Fig3 shows). However, it was found that the system failed toproperly correct errors in some specific cases:The system cannot discriminate when the user is playingwith one hand or another, but instead makes a globalcorrection of both hands at the same time.It may be possible that a mistake when playing a figureis assumed as a a mistake in the note played. I.e., if themusician is playing a long note with his left hand andreleases it slightly before time, the lack of its spectralcontribution could affect the detection of the notes playedwith the right hand.V. CONCLUSIONSIn this paper, we have presented a system that analyzes apolyphonic piano recording and assesses the correctness of thepiece played. The system automatically segments the musicalsignal, identifying the notes played and their duration, as wellas indicating the type of mistakes the musician performedin the piece recorded. The tests conducted showed that thesystem has a very good performance and does indeed fulfilits purpose as a support tool in the learning processes. Thesystem designed works for any kind of piano and does notrequire prior training of any type.ACKNOWLEDGEMENTSThis work has been funded by the Ministerio de Economia yCompetitividad of the Spanish Government under Project No.TIN2010-21089-C03-02 and by the Ministerio de Industria,Turismo y Comercio under Project No. TSI-090100-2011-25.REFERENCES[1] Ajjan, H. and Hartshorne, R. “Investigating faculty decisions to adoptWeb 2.0 technologies: Theory and empirical testsc,” The Internet andHigher Education vol. 11, no. 2, pp. 71-80, 2008.[2] Condie, R. and Munro, B. “The impact of ICT in schools: Landscapereview,” British Educational Communications and Technology Agency(BECTA), 2007.[3] Harrison, C, Comber, C, Fisher, T, Haw, K, Lewin, C, Lunzer, E, McFar-lane, A, Mavers, D, Scrimshaw, P, Somekh, B and others. “ImpaCT2: Theimpact of information and communication technologies on pupil learningand attainment,” British Educational Communications and TechnologyAgency (BECTA), 2002.[4] Ajjan, H. and Hartshorne, R. “Sowing the seeds for a more creativesociety,” Learning and Leading with Technology vol. 35, no. 4, pp. 18-22,2007.[5] Bello J.P, Daudet L, Abdallah S, Duxbury C, Davies M, Sandler M.B.“A tutorial on onset detection in music signals,” IEEE Transactions onSpeech and Audio Processing, vol.13 , no. 5, pp. 1035-45, IEEE. 2005.[6] Boogaart, C.G and Lienhart, R. “Note onset detection for the transcriptionof polyphonic piano music, ” IEEE International Conference on Multi-media and Expo, ICME 2009, pp. 446-449, 2009.[7] Benetos, E and Dixon, S. “Polyphonic music transcription using noteonset and offset detection, ” IEEE International Conference on Acoustics,Speech and Signal Processing (ICASSP), pp. 37-40, 2011.[8] Barbancho I, Barbancho A.M, Jurado A, Tardo´n L.J. “Transcription ofpiano recordings,” Applied Acoustics, vol.65 , no. 12, pp. 1261-66, 2005.[9] Barbancho A.M, Barbancho I, Soto B, Tardo´n L.J. “SIC receiver forpolyphonic piano music,” In Proc. ICASSP, pp. 377-380, 2011.

Correction system for polyphonic piano recordings

Abstract

Similar works

Full text

Available Versions

Repositorio Institucional Universidad de Málaga