Musicians' ability has an important role in music performance. What does it happen in computer music playing? In this paper a connessionist approach to the problem of computer music performance is proposed. A hybrid system for computer music automatic performance is showed. In this system, two sets of symbolic and sub-symbolic rules act in co-operation. We propose also some neural nets' models, which learn the performing style of a professional pianist. Some models and applications of neural nets (NNs), that we built, are discussed

Bresin, Roberto

De Poli, Giovanni

Vidolin, Alvise

Archivio istituzionale della ricerca - Università di Padova

Original Citation:Symbolic and Sub-symbolic Rules System for Real Time Score performanceInternational Computer Music AssociationPublisher:Published version:DOI:Terms of use:Open Access(Article begins on next page)This article is made available under terms and conditions applicable to Open Access Guidelines, asdescribed at http://www.unipd.it/download/file/fid/55401 (Italian only)Availability:This version is available at: 11577/3313731 since: 2019-11-05T17:22:30ZUniversità degli Studi di PadovaPadua Research Archive - Institutional RepositorySymbolic and sub-symbolic rules system for real time score performance Roberto Bresin, Giovanni De Poli, Alvise Vidolin C.S.C.- Padua University- via Gradenigo 6/a, 35131 Padova- Italy E.mail: depoli@dei.unipd.it- Tel.: +39 (0)49 8287631- Fax: +39 (0)49 8287699 ABSTRACT Musicians' ability has an important role in music performance. What does it happen in computer music playing? In this paper a connessionist approach to the problem of computer music performance is proposed. A hybrid system for computer music automatic performance is showed. In this system, two sets of symbolic and sub-symbolic rules act in co-operation. We propose also some neural nets' models, which learn the performing style of a professional pianist. Some models and applications of neural nets (NNs), that we built, are discussed. 1. INTRODUCTION Musicians, according to the instruments they play, execute intensity, durational and timbric deviations, since the traditional musical notation is not suffice enough to the composer's intentions. As a matter of fact, music performed according to the metronomic symbols sounds extremely mechanic and dull to the listener. These deviations determine the performing style and the ability of a musician in contrast to another. The problem of performance is a crucial aspect in computer music, where the computer itself is the performer. The execution, obtained with a literal translation of the classical performing symbols of a score, leads to an extremely mechanic performance. So one feels the need to performe in a warmer and more human way. In the field of musical performance many researches have been made in the past years (Seashore, 1938; Repp, 1990; Clynes, 1983; De Poli, Irone and Vidolin 1990; Sundberg, 1991; Friberg, 1991). In particular Sundberg and his eo-workers at the KTH in Stockholm built a symbolic rules' system using the analysis-by-synthesis method. The purpose of these rules is to change a music score into an acceptable performance. The resulting deviations from the rules are additive because each note can be processed by several rules. These deviations made from each rule, are successively added to the parameters of that note. With this method expert musicians and teachers were asked to formulate some performing rules. These rules, from the suggestions of an expert musician, who judged whether the adding effect of the single rule was pleasant or not, were applied in sequence during the performance . The performance is obtained through the sum of time, sound and timbre deviations due to the simultaneous application of numerous symbolic rules. But all these systems aren't able to achieve a good performance since symbolic rules can't consider sense-motory activities of the performer. In the present work, we start from the results of Sundberg and we build a hybrid system, where two sets of symbolic and sub-symbolic rules act in co-operation. The idea is to integrate the results from some small neural nets (NNs) controlled by symbolic rules. 2. A HYBRID SYSTEM If we compare our NNs with a physiological system we could say that the input neurons correspond to the optic nerves of the pianist and the output neurons corresponds to the hands of the pianist himself. The cause is the score reading (i.e. the stimulus to the optic nerves) and the effect is the mechanic action (the hands of the pianist). The two most important parts of the hybrid system are the symbolic rules and the sub-symbolic rules. The symbolic rules are divided in two sub-sets: decision rules, and performing rules. Decision rules are necessary to select a particular net or a particular input to the net according to the particular musical situation. For instance, supposed there is a refrain in the musical score, the rules will choose a net to perform the refrain the first time and another net to perform it the second time. Another example could be a NN that an input node representing an enphasis parameter (see the K parameter of the Sundberg's rules): we need some decision rules to determin which is the preferred value of this parameter. We left some performing rules in the symbolic rules' set since they reflect some conscious decision of the performer (e.g. phrasing rules). The performance is also made by non-formalizing rules (sub-symbolic rules) depending, for example, from the performer's sensibility and from his sense-motory activities. We suppose that these kinds of behaviors could be modeled by NNs. It is possible to train the NNs with particular inputs and outputs. At first we used the standard feed-forward NN with the back-propagation algorithm, using the same NNs' structures described in the next section (see figure 3.1). The outputs of the NN were the time and loudness deviations, and the inputs depended from the particular experiment and from the particular architecture of the NN. The score is analysed by a set of symbolic rules, which will produce a particular output (0UT2 in figure 2.1). This output, after a codify, can be "read" by aNN. The net itself, i.e. the sub-symbolic rules, gives an output (OUTl in figure 2.1). This output will be added to OUT2. The resulting sum is applied to the nominal score to produce the "out score", that is the score produced by the rules. The new score is sent, through the MIDI interface, to the synthetizer. The analysis of the score, by mean of the symbolic rules, allows to determine when the computer is performing a particular "musical pattern". For each of these patterns, the system chooses a particular NN or a particular value of the K parameter, that is an input to the NN. 3. LEARNING FROM SUNDBERG' PERFORMANCE RULES The first problem in using NNs is the choice of inputs, which must be easy to derive (in the case of real time performance it is important to have the inputs without any futher calculation) and not numerous (in order to have few input neurons), and the choice of the outputs, i.e. the parameters we want to influence. Most of the rules include a parameter K used as weight to change their influence. Since the Sundberg's rules' system works well, we decided to use the parameters of these rules as inputs to our NNs and we taught to the net some of those rules acting at a microscopic level. It has been trained the output of the net with the results (i.e. time or loudness deviations are taught according to the particular net) obtained from application of a subset of symbolic rules to the examining score, using the back-propagation algorithm. The rules give the deviation for each note and this deviation is used to train the NN. To train aNN like that in figure 3.1 giving in input few significant patterns, one must take two minutes using a P.C. with a 80387-33MHz. This is a good result if you think that a beginner plays music like these networks after at least one year of practice. The net in figure 3.1 is the base NN we used: it has four input's nodes and one output node. The input's terms are extracted from Sundberg's symbolic rules except for the third input node: this node is necessary in order to avoid ambiguities while teaching the "leap tone duration" rule (Friberg, 1991). We trained these NNs with four rules: the "durational contrast" rule, the "melodic charge" rule, the "high loud" rule, and the "leap tone duration" rule. A further step was to train three different nets with the same structure of the one in figure 3.1, but we weighed the rules with three different values of the K parameter. So we built three networks corresponding to the following values ofK: K=l, K=5, and K=9. The net with K=l gives variations, that are difficult to hear, and the net trained with K=9 gives emphasized time variations. We built also another net, with five input neurons. The first four are the same of the previous experiments and the fifth has in input the K parameter. With such a network, in a real time performance, it is possible, changing the value of K, to obtained various time deviations: one can change from a "baroque performing style" (K=l) to a " romantic style" (K=9). The trained NN shows good generalizations' properties: it behaves well with new input's patterns (i.e. never taught inputs), and it gives better results when the Sundberg's rules would give an excessive output (due to the additivity of the rules' system). 4. LEARNING A PERFORMING STYLE A further step was to implement NNs, which can learn the performing style of a professional pianist, by means of his own performance. We tried to built such nets starting from the performance of the third movement of the Beethoven's piano sonata Op.31 No. 3 performed by the pianist Piero Pittari with a synthesizer. Analysing the metronome variations of the bar's quarters, we noticed a good overlap with the Repp's data (Repp, 1990), which are obtained from the analysis of the performances of nineteen famous pianists. The characteristic "V" pattern of some bars shows the so called "Beethoven pulse" (Clynes, 1983). In this "pulse" the second quarter bar is faster than the first one and the third one is slower than the first one. The first experiments were made using the same net of figure I. This time, the values, to train the output of the net, were extracted from the performance of Pittari, instead to use the Sundberg's rules to produce the training's outputs. The structure of this net was not sufficient to achieve a good result, because the input's neurons were few to define the performing style of a pianist. So, to solve this problem, we introduced another input neuron, that consideres the edges of the phrase's ligatures. Usally, in these edges, there are some "rallentando" or "breaths". We also considered two different nets for the two refrains of the minuetto, because the pianist plays in two different ways the refrains. This difference is necessary to avoid a boring performance. The obtained results are good: some listeners recognized, in the minuetto played by the net, some characteristics of the style of the pianist Pittari. Looking at the flow of the quarters of the minuetto performed by the net, one can observe that there are the characteristic "V" patterns in twenty bars. These patterns do not always correspond to that of the original performance of the pianist, but they show that the trained NN plays with a "Beethoven pulse". In any case, it is interesting to observe that the style of the pianist, made of marked time deviations between the first and the third quarter, is preserved. Comparing these results with the output of the previous explained hybrid system, one can see that the output of the net, that is trained with the style of the pianist, is more "interpreted". To obtain better results, we chose a more complex model of net (figure 4.1): the ecological NN (Cecconi et al. , 1990). This net considers the stimuli coming from the environment (net's inputs) in theN cycle as a function both of the characteristics of the environment and of the action of the net itself (net's output) at the N-1 cycle. The net we used, is showed in figure 10: there is a new input node, that considers the previous output of the net. This is a net with feed-back. In this model of net, the new input neuron could represents the acoustic nerves, i.e. the pianist listens to himself playing and acts accordingly. The ecological net gave a better result with respect to the NN without feed-back: comparing the outputs of the two NNs with the durations of the pianist's performance, the ecological NN gives a three times smaller error. 5. CONCLUSIONS It is important to point out that, in all the described experiments, with a training set of only two or three bars (which are chosen among those with the most difficult passages), we have good performances for any number of notes and for never taught situations. The good results of this research confirm the initial hypothesis with respect to the effectiveness of the use of artificial NNs in the musical performance. Such results suggest a further development of the system. It is important to outline that the nets, built in this research, allow a real time performance for every score, by just knowing some foundamental paramenters, that are the nominal duration of the note, its nominal intensity, and its position in the score. This procedure is new if compared to the previous approaches of computer performance, consisting either in a "manual" adjustment of the performing paramenters of each note, or in using an expert system, that sets these parameters by means of symbolic rules. There are two basic differences between the performance with NNs and the two above mentioned methods: the former is that the performance takes place in real time, once the values of duration, intensity, and position of each note are known, the latter, and more important one, is that the net's training is indipendent from the context. An important result is the possibility of letting the NN perform any score in real time and with the same style, that was previously taught to the net in a short time. Another achievement is the "smoothing" of output deviations compared with the exaggerated ones, that are obtained by Sundberg's symbolic rules adding action. The most important application of the present work is to perform computer generated musics, since the computer is not able to set, without any "human help", the time, intensity, and timbre deviations, which, on the contrary, are distinctive characteristics of the performers' ability. In this direction, a possible use of our system is the automatic performance of computer generated scores: the computer automatically introduces both duration and intensity micro-deviations, in order to obtain a more "melodious" performance. In particular, we chose the NNs we described in the previous pages, which allow to change in real time the K parameter of the Sundberg's rules, and we applied these nets to the output of a "markovian" source of melodies (Bresin, Manduchi 1989), consisting in an automatic improvisation on a previously played theme. It is possible to change the K parameter during the performance and to vary, in every istant, from a "cold" performance (low values of K) to a "romantic" one (high values of K). The results of the present work may also have a commercial application: one could think to store NNs in the synthetizer's ROM, giving the possibility of automatically "interpreted" performances. 6. REFERENCES Bresin, R., Manduchi,R. "Una sorgente di melodie con controllo di entropia", Proceedings of the VIll Colloquium on Musical Informatics, Cagliari, pp. 213-215, October 989 Clynes, M. "Expressive microstructures in music, linked to living qualities". In J. Sundberg (Ed.), Studies of music performance, Stockholm: Royal Swedish Academy of Music, Publication No. 39, pp. 76-181, 1983 Cecconi F., Parisi D. "Learning to predict the consequences of one's own actions", in R. Eckmiller et al. (cur.) Parallel processing in neural systems and computers, Amsterdam: North-Holland, 1990 De Poli, G., Irone, L., Vidolin, A. "Music score interpretation using a multilevel knowledge base", Interface, vol. 19, pp. 137-146, 1990 Friberg, A. "Generative Rules for Music Performance: A Formal Description of a Rule System", Computer Music Journal, vol. 15, No. 2, pp. 56- 71, Summer 1991 Repp, B.H. "Patterns of expressive timing in performances of Beethoven minuett by nineteen famous pianists", J. Acoust. Soc. Am., vol. 88, No. 2, pp. 622-641, August 1990 Repp, B.H. "Further Perceptual Evaluations of Pulse Microstructure in Computer Performances of Classical Piano Music", Music Perception, vol. 8, No. 1, pp. 1-33, 1990 Rumelhart, D.E., McClelland, J.L. Parallel Distributed Processing, vol. 1, Cambridge: MIT Press, chapter 8, 1988 Seashore, C.E. Psychology of Music, New York: McGraw-Hill, 1938 Sundberg, J., Friberg, A., Fryden, L. "Rules for automated performance of ensemble music", Contemporary Music Review, Vol. 3, pp. 89-109, 1989 Sundberg, J. et al. "Performance Rules for Computer-Controlled Contemporary Keyboard Music", Computer Music Journal, vol. 15, No. 2, pp. 49-55, Summer 1991 Sundberg, J., Thompson, W.F., Friberg, A., Fryden, L. "Use of rules for expression in the performance of melodies", Psychology of music, vol. 17, No. 1, pp. 63-82, 1989 L\DR n-1 Nominol Duration figure 2.1 Melodic Charge leap Presence figure 4.1 Semitons Start/End in a leap Phrase Nominol Duration Variation of Duration leap Semitones Melodic Charge Presence in o leop figure 3.1 

Symbolic and Sub-symbolic Rules System for Real Time Score performance

Abstract

Similar works

Full text

Available Versions

Archivio istituzionale della ricerca - Università di Padova