Continuous MIDI controllers commonly output their position only, with no influence of the performative energy with which they were set. In this paper, creative uses of time as a parameter in continuous controller mapping are demonstrated: the speed of movement affects the position mapping and control output. A set of SuperCollider classes are presented, developed in the author’s practice in computer music, where they have been used together with commercial MIDI controllers. The creative applications employ various approaches and metaphors for scaling time, but also machine learning for recognising patterns. In the techniques, performer, controller and synthesis ‘intra-act’, to use Karen Barad’s term: because position and velocity are derived from the same data, sound output cannot be predicted without the temporal context of performance

Nystrom, E.

English

City Research Online

              City, University of London Institutional RepositoryCitation: Nystrom, E. ORCID: 0000-0003-3628-540X (2020). Intra-Actions: Experiments with Velocity and Position in Continuous Controllers. Poster presented at the NIME 2020, 21-25 Jul 2020, Birmingham, UK. This is the published version of the paper. This version of the publication may differ from the final published version. Permanent repository link:  https://openaccess.city.ac.uk/id/eprint/24994/Link to published version: Copyright and reuse: City Research Online aims to make research outputs of City, University of London available to a wider audience. Copyright and Moral Rights remain with the author(s) and/or copyright holders. URLs from City Research Online may be freely distributed and linked to.City Research Online:            http://openaccess.city.ac.uk/            publications@city.ac.ukCity Research Online367Intra-Actions: Experiments with Velocity and Position in Continuous Controllers   Erik Nyström  City, University of London  Northampton Square,  London, EC1V 0HB, UK  Erik.Nystrom@city.ac.uk     ABSTRACT  Continuous MIDI controllers commonly output their position only, with no influence of the performative energy with which they were set. In this paper, creative uses of time as a parameter in continuous controller mapping are demonstrated: the speed of movement affects the position mapping and control output. A set of SuperCollider classes are presented, developed in the author’s practice in computer music, where they have been used together with commercial MIDI controllers. The creative applications employ various approaches and metaphors for scaling time, but also machine learning for recognising patterns. In the techniques, performer, controller and synthesis ‘intra-act’, to use Karen Barad’s term: because position and velocity are derived from the same data, sound output cannot be predicted without the temporal context of performance.  Author Keywords Synthesis, Controllers, Performance, Machine Learning CCS Concepts •Applied computing~Arts and humanities~Performing arts•Applied computing~Arts and humanities~Sound and music computing 1. INTRODUCTION The techniques discussed here are derived from my ongoing artistic practice in computer music, and were featured in the works Texton Mirrors (2018/19, most recently performed at Ars Electronica Festival 2019) and Intra-action (commissioned by and premiered at NEXT Festival 2019). These are both live computer music works which incorporate improvisation and algorithmic processes and conceptually draw on posthumanist philosophy [2, 9] by exploring how human action and digital processes can form a single technological system. Intra-action is influenced by Karen Barad’s agential realist philosophy and the concept of ‘intra-action’, according to which objects and phenomena do not precede their encounters [2]. In the work, sound synthesis systems are structured both by capturing performance characteristics and by self-organised, agent-based models, where processes respond to one another. The time-dependent techniques are an important element as their inherent conflict between movement and position demonstrate a technical system whose behaviour cannot be predicted without physical and temporal context: this coupling physical materiality, abstract information and sound constitutes an intra-action in itself.  Knobs and faders are typically designed to control sound by static coordinate-positions in a parameter range which is unaffected by the performative energy of the player. Although one can sweep continuously at any speed of hand movement, the speed itself does not affect the mapped output. The techniques developed here constitute one way of extending the control system so that the sound cannot be predicted by controller position settings alone. Instead, hand movements become an integral part of the technology of sound-making, as velocity and position are entangled in the same gesture. While this does not literally add physical sensitivity to the hardware interface, it adds a physically-driven, sensorimotor-dependent, constraint to the software mapping of any basic hardware, such as MIDI controllers.  2. BODY, CONTROL, SOUND Performance sensitivity and physicality is a core topic in computer music research which often draws upon embodied cognition and phenomenology (1, 7, 12, 14, 16). Many discussions of performance and mapping look to instrumental acoustic music for direction (11, 14). In Paine’s [14] framework for ‘embodiment relations’ based on instrumental performers’ accounts, speed emerges naturally (bow speed, air velocity). The solution to incorporating such factors in computer music is often physical interfaces that emulate instruments and respond to related bodily actions. I believe we should be careful when modelling computer music on acoustic instrumental musicianship, as we then may fail to embrace the virtual, technological nature of electronic sound. Instead, I prefer introducing physicality through digital means, replacing ‘true’ physicality with technologically generated multimodal constraints. An important work of related research is Newton Armstrong’s [1] which outlines criteria for enactive performance as situated, timely, engaging, multi-modal, and emergent. Armstrong’s work is relevant here because his design solutions are predicated on music technological archetypes such as knobs and buttons rather than novel interfaces, and because it integrates these closely to non-linear sound synthesis models. Also relevant are Bowers et al [3], who draw upon Barad in the aim of designing electronic instruments that ‘meet us halfway’. Their work under the ‘self-consciously provocative’ theme ‘one knob to rule them all’ (p. 433) featured the design of the Hyperpot which added capacitive sensing to a rotary controller, and used the output creatively in a variety of contexts, stating that “we find devices which create challenges for us more performable than those which simply bend to our will.” (p. 438). The experiments in [10] showed that multiparametric interfaces which link control inputs in non-obvious manners were preferred by performers as they allow for a gestural, non-parametric relationship with sound.   However, there is no mention of speed-dependent control in either of the above examples. In my techniques, the velocity sensitivity and non-linearity of mappings inherently produce morphological properties in sound. Used creatively, this linking can forge a multimodal ‘added value’ [4], as sound output influences impressions of, for instance, force, friction, and elasticity in the meeting of hand movement and controller. 3. MAPPING MODELS The simple operation of measuring the time delta between every current and previous value generated by controller movement is the starting point. This value is then used in the mapping, either alone or with control position. The mappings all apply some form of divergent mapping (one-to-many), sometimes in conjunction with convergent mapping (many-to-one) [13], where several controllers are mapped differently to the same 368sound, to produce conflict. The below-mentioned classes, developed in the SuperCollider programming environment, can be downloaded here: https://github.com/postnature/delta-control 3.1 Direct Velocity Mapping The most basic approach is based on mapping the control deltas directly to a sound parameter. If this is mapped to the gain of distortion of an oscillator, for instance, the timbre will become thicker (or ‘warmer’) the faster the controller is moved: the sound suggests an increase in energy, approximately corresponding to the energy of performance. In the class DeltaThresh this can be augmented with a velocity threshold, above and below which mapping is different. This allows for critical transitions where sound radically changes beyond a threshold of energy, like a clipping signal or a shattering surface.   3.2 Chase: Velocity and Position Controller velocity and position can be merged through the metaphor of ‘chasing’, every knob movement being an attempt to reach a value associated with a position or distance: the faster the controller is turned, the further along a parameter space its position will be mapped. Two approaches are used here.  3.2.1 ChaseValue The first, ChaseValue, is based on the principle that only a given maximum velocity of controller movement will return the actual value of the controller position: any lower velocities will be scaled in the range from minimum to current position. For example, if the knob position is equivalent to MIDI value 70, then the output value will instead be somewhere between 0 and 70 depending on velocity. This means that smooth sweeps are difficult to achieve, and discontinuous, jumpy shapes are more likely to result. Speed-dependent continuity is like a metaphor for gravity: with sufficient momentum the successive values ‘fly’ in a smooth contour; at slower movement, they keep dropping to the ground (the minimum). 3.2.2 ChaseRatio The second, ChaseRatio, is a process that measures not the actual controller position, but only whether the position is above or below the previous. Depending on whether movement is positive or negative it is multiplied or divided by a value which is the product of a specified ratio and the speed of movement. A higher ratio specified will make the controller more sensitive to velocity. The effect of this type of mapping is that fast movements produce a course tuning and slow movements a fine tuning of a parameter. As a consequence, one can use the whole controller range to explore the surroundings of a specific value with slower movements, but also drop or raise the parameter with a more sudden move. The accuracy of this is entirely in the hand of the performer and cannot be approximated by position alone – it is thus an entirely embodied interface. Because any new value is relative to the previous, rather than corresponding to a fixed range, the output depends on an emergent accumulated history of movements. In Intra-action, these techniques are used for controlling parameters such as oscillator frequency and distortion, and for a non-standard method where successive wave form segments derived from controller movement are directly fed into an array which forms an oscillating wave. 3.3 Neural Networks While the above techniques concern mappings of individual velocities, collections of values can be used with machine learning for gesture recognition. This is powerful in cases where one wants to link sounds with more specific gestures: for instance, I found that mapping the recognition of a slow wide sweep to the triggering of a graduated-continuant morphology [15] links a sustained physical effort to a longer-duration sound, producing a sensorimotor and sound-perception correlate. In Intra-action, I use the NeuralNet class [6], trained to recognise sequences of time deltas. Each control change feeds a new time value into an array (which simultaneously drops its oldest value) and calculates an output on the basis of this array.  4. CONCLUSION Using these techniques, precision becomes a bodily skill: sound is located in movement rather than in fixed position coordinates. In Armstrong’s terms, the processes described here are situated, because the parameter spaces change and modify the context of performance agency; they are timely for the obvious reason that time is central; they are engaging because the mappings require bodily velocity; they are multimodal because sound constrain and augment physical control; they are emergent because sound output cannot be predicted on a single position, but depend on accumulated velocities. The research itself is intra-active, as creative practice is entangled with technology. 5. ACKNOWLEDGEMENTS Thanks to NEXT Festival for commissioning Intra-action.  6. REFERENCES [1] N. Armstrong, An Enactive Approach to Digital Musical Instrument Design, Saarbrücken: Av Akademikerverlag. 2012. [2] K. Barad, Meeting the Universe Halfway, London: Duke University Press. 2007. [3] J. Bowers et. al. One Knob to Rule Them All: Reductionist Interfaces for Expansionist Research, Proc. of NIME 2016, pp. 433-438. [4] M. Chion, Audio-Vision. New York: Columbia University Press. 1994. [5] P. Cook, Principles for Designing Computer Music Controllers, Proc. of NIME 2001, pp. 1-6. 2001. [6] N. Collins, NeuralNet, SuperCollider. GNU GPL 2007. [7] P. Dalhstedt, Action and Perception: Embodying Algorithms and Extended Mind, Oxford Handbook of Algorithmic Music pp. 41-66, 2018. [8] G, E. Harnett and C. Goudeseoune, Performance Factors in Control of High-Dimensional Spaces,  [9] N. K. Hayles, How We Became Posthuman. Chicago: University of Chicago Press. 1999.  [10] A. Hunt and M. M. Manerley, Mapping Performer Parameters to Synthesis Engines, Organised Sound 7(2) pp. 97-108, 2002. [11] A. Hunt, and R. Kirk, Mapping Strategies for Musical Performance, Trends in Gestural Control of Music, M.M. Wanderley and M. Battier, eds. Ircam Centre Pompidou, pp. 231-258, 2000. [12] B. Ostertag, Human Bodies, Computer Music [13] J. B. Rovan, M. M. Wanderley, S. Dubnov, and P. Depalle, Instrumental Gestural Mapping Strategies as Expressivity Determinants in Computer Music Performance, Proc. of the Kansei Workshop, Genova, pp. 68-73 [14] G. Paine, Gesture and Morphology in Laptop Performance, The Oxford Handbook of Computer Music pp. 214-232. 2011. [15] D. Smalley, Spectromorphology: Explaining Sound Shapes. Organised Sound 2(2), pp. 107-26, 1997. [16] D. Wessel, and M. Wright, Problems and Prospects for Intimate Musical Control of Computers, Computer Music Journal, 26:3, pp. 11-22, Fall 2002  [17] M. Wright, D. Wessel, and A. Freed,	New	Musical	Control	Structures	from	Standard	Gestural	Controllers,	Proc.	of	the	ICMC,	1997

Intra-Actions: Experiments with Velocity and Position in Continuous Controllers

Continuous MIDI controllers commonly output their position only, with no influence of the performative energy with which they were set. In this paper, creative uses of time as a parameter in continuous controller mapping are demonstrated: the speed of movement affects the position mapping and control output. A set of SuperCollider classes are presented, developed in the author's practice in computer music, where they have been used together with commercial MIDI controllers. The creative applications employ various approaches and metaphors for scaling time, but also machine learning for recognising patterns. In the techniques, performer, controller and synthesis 'intra-act', to use Karen Barad's term: because position and velocity are derived from the same data, sound output cannot be predicted without the temporal context of performance

Nyström, Erik

NEUROSURGERY ENTHUSIASTIC WOMEN SOCIETY

https://openaccess.city.ac.uk/id/eprint/24994/1/nime2020_paper71.pdf

Intra-Actions: Experiments with Velocity and Position in Continuous Controllers

Abstract

Similar works

Full text

Available Versions

City Research Online

NEUROSURGERY ENTHUSIASTIC WOMEN SOCIETY