Deep Visual Instruments: Realtime Continuous, Meaningful Human Control over Deep Neural Networks for Creative Expression

In this thesis, we investigate Deep Learning models as an artistic medium for new modes of performative, creative expression. We call these Deep Visual Instruments: realtime interactive generative systems that exploit and leverage the capabilities of state-of-the-art Deep Neural Networks (DNN), while allowing Meaningful Human Control, in a Realtime Continuous manner. We characterise Meaningful Human Control in terms of intent, predictability, and accountability; and Realtime Continuous Control with regards to its capacity for performative interaction with immediate feedback, enhancing goal-less exploration. The capabilities of DNNs that we are looking to exploit and leverage in this manner, are their ability to learn hierarchical representations modelling highly complex, real-world data such as images. Thinking of DNNs as tools that extract useful information from massive amounts of Big Data, we investigate ways in which we can navigate and explore what useful information a DNN has learnt, and how we can meaningfully use such a model in the production of artistic and creative works, in a performative, expressive manner. We present five studies that approach this from different but complementary angles. These include: a collaborative, generative sketching application using MCTS and discriminative CNNs; a system to gesturally conduct the realtime generation of text in different styles using an ensemble of LSTM RNNs; a performative tool that allows for the manipulation of hyperparameters in realtime while a Convolutional VAE trains on a live camera feed; a live video feed processing software that allows for digital puppetry and augmented drawing; and a method that allows for long-form story telling within a generative model's latent space with meaningful control over the narrative. We frame our research with the realtime, performative expression provided by musical instruments as a metaphor, in which we think of these systems as not used by a user, but played by a performer

tiNNbre: a timbre-based musical agent

In this paper, we present tiNNbre, a generative music prototype-system that reacts to timbre gestures. By timbre gesture we mean a sonic (as opposed to body) gesture that mainly conveys artistic meaning through timbre rather than other sonic properties. The system is designed and developed to be used in free improvisation and composition. Our prototype is powered by a neural network trained using a supervised learning approach on a set of sonic gestures representing the stimulus (or input) and a correspondent set of sonic gestures representing the reactions (or output). This model is then used to explore and generate new musical material. Here, we present an informal evaluation of our system, based on two use cases. In the first, the system is trained using MFCC analysis, while the second uses Constant-Q Transform spectrum. Participants were asked to rate our system's generated audio, particularly in respect of timbre. Results showed that although tiNNbre has been tested offline, it fosters timbre-rich interaction with the sonic materials and the co-creation process. Broadly speaking, musicians preferred the results obtained using the second system other than for purely percussive gestures

Designing and evaluating the usability of a machine learning API for rapid prototyping music technology

To better support creative software developers and music technologists' needs, and to empower them as machine learning users and innovators, the usability of and developer experience with machine learning tools must be considered and better understood. We review background research on the design and evaluation of application programming interfaces (APIs), with a focus on the domain of machine learning for music technology software development. We present the design rationale for the RAPID-MIX API, an easy-to-use API for rapid prototyping with interactive machine learning, and a usability evaluation study with software developers of music technology. A cognitive dimensions questionnaire was designed and delivered to a group of 12 participants who used the RAPID-MIX API in their software projects, including people who developed systems for personal use and professionals developing software products for music and creative technology companies. The results from the questionnaire indicate that participants found the RAPID-MIX API a machine learning API which is easy to learn and use, fun, and good for rapid prototyping with interactive machine learning. Based on these findings, we present an analysis and characterization of the RAPID-MIX API based on the cognitive dimensions framework, and discuss its design trade-offs and usability issues. We use these insights and our design experience to provide design recommendations for ML APIs for rapid prototyping of music technology. We conclude with a summary of the main insights, a discussion of the merits and challenges of the application of the CDs framework to the evaluation of machine learning APIs, and directions to future work which our research deems valuable

CIRCUS 2001 Conference Proceedings: New Synergies in Digital Creativity. Conference for Content Integrated Research in Creative User Systems

CIRCUS (Content Integrated Research For Creative User Systems) was an ESPRIT Working Group, originally set up in 1988 as one of the very last additional actions in Framework 4, under DG III. Its purpose was to develop models for collaborative work between artists (the term here used in its widest sense) and technologists (ditto) and to promote these models by whatever means available. While some have criticised this aim as implicitly promoting a 1950s agenda of building bridges across C.P. Snow’s ‘two cultures’, there is no such intention here, rather that technology, particularly computer and communications technology (ICT) , is irresistibly intruding into what is normally thought of as creative work (and so practised by artists) and that, like any new technique, this has to be understood by its potential practitioners in terms of its true strengths and limitations. The specific problem that computer technology poses is that it is in principle malleable to such an extent that the limitations on its form and functionality are still barely understood, yet the people charged with the task of making the technology available have little or no understanding of the needs of creative users. What the artist usually sees is a tool which is in principle capable of being harnessed to creative ends but in practice resists being so applied. Quite often the tool is shaped more by blind economic forces than by a clear response to a specific, here creative, need. CIRCUS came into existence as a forum in which both artists and technologists could work out how best to play to the strengths of ICT and how to apply both creative and technological solutions (possibly both together) to its limitations. In particular the then new Framework V programme invited projects in such areas as new media but required them to be addressed in essentially the same old way, by technologists working towards commercialisation. The only obvious exception to this was in the area of cultural heritage which, incidentally, CIRCUS was also capable of reviewing. The scope for effective participation by artists was thus limited by an essentially technological agenda although everybody at the time, the participants of CIRCUS and programme managers in DG III, believed that we could do far better than this, and to develop new models of working which could inform the nature of Framework VI or even the later stages of F V. It is fair to say that everyone involved was excited by the idea of doing something quite new (and iconoclastic), not least the expanding of the expertise base on which future Frameworks could draw. It is also fair to say that, while not ultimately wholly original, the CIRCUS agenda was an ambitious one and the WG has had a chequered history peppered with misunderstandings perpetrated by the very people who might have thought would give the WG their strongest support. The CIRCUS idea has been aired before, specifically at the University of Illinois at Urbana- Champaign, the MIT Media Lab (and its imitators), and a recent IEEE forum. However a near total change in participation, fuelled by natural migration and a switch to DG XIII, has resulted in the CIRCUS agenda being restarted on at least one occasion and a fairly regular questioning of the principles on whose elucidation we are engaged. While this is no bad thing in principle, in practice we haven’t learned anything new from these periodic bouts of self-examination other than a reinforcement of the values our goals. On the other hand it is evident that we have made progress and have moved on a long way from where we started. A recent experience of a workshop whose agenda appeared to be to form another version of CIRCUS, this time with an overwhelmingly technological (DG III) membership, demonstrates they have a CIRCUS-worth of work to do before they will have reached where we are now. (Foreword of CIRCUS for Beginners

Musical Haptics

Haptic Musical Instruments; Haptic Psychophysics; Interface Design and Evaluation; User Experience; Musical Performanc