Nonlinear Dynamical Processes in Musical Interactions: investigating the role of nonlinear dynamics in supporting surprise and exploration in interactions with digital musical instruments

Nonlinear dynamical processes play a central role in many acoustic instruments, yet they rarely feature in digital instruments, and are little understood from an interaction design perspective. Such processes exhibit behaviours that are complex, time-dependent, and chaotic, yet in the context of acoustic instruments can facilitate interactions that are explorable, learnable and repeatable. This suggests that these processes merit deeper investigation for digital music interaction design. Two studies are presented which investigate user interaction with nonlinear dynamical musical tools. A lab-based study used four purpose-built digital musical instruments to test interaction designs featuring nonlinear dynamical processes. Evaluations with 28 musicians demonstrated the potential for these processes to provoke creative surprises, and support exploration without a corresponding loss of control. A subsequent ethnographically-informed study with 24 musicians linked these findings to a mode of engagement which we term ‘edge-like interaction’. Edge-like interactions draw on the complex, unpredictable behaviours found in nonlinear dynamical processes close to critical thresholds, facilitating creative exploration. The two complementary studies provide evidence both for the existing importance of nonlinear dynamical processes in musical interactions with acoustic interactions, and their potential for deployment in the development of new creative digital technologies, musical or otherwise

A framework for digital watercolor

This research develops an extendible framework for reproducing watercolor in a digital environment, with a focus on interactivity using the GPU. The framework uses the lattice Boltzmann method, a relatively new approach to fluid dynamics, and the Kubelka-Munk reflectance model to capture the optical properties of watercolor. The work is demonstrated through several paintings produced using the system

Real-time simulation of watery paint

Existing work on applications for thin watery paint is mostly focused on automatic generation of painterly-style images from input images, ignoring the fact that painting is a process that intuitively should be interactive. Efforts to create real-time interactive systems are limited to a single paint medium and results often suffer from a trade-off between real-timeness and simulation complexity. We report on the design of a new system that allows the real-time, interactive creation of images with thin watery paint. We mainly target the simulation of watercolor, but the system is also capable of simulating gouache and Oriental black ink. The motion of paint is governed by both physically-based and heuristic rules in a layered canvas design. A final image is rendered by optically composing the layers using the Kubelka-Munk diffuse reflectance model. All algorithms that participate in the dynamics phase and the rendering phase of the simulation are implemented on graphics hardware. Images made with the system contain the typical effects that can be recognized in images produced with real thin paint, like the dark-edge effect, watercolor glazing, wet-on-wet painting and the use of different pigment types