Functional Scaffolding for Musical Composition: A New Approach in Computer-Assisted Music Composition

While it is important for systems intended to enhance musical creativity to define and explore musical ideas conceived by individual users, many limit musical freedom by focusing on maintaining musical structure, thereby impeding the user\u27s freedom to explore his or her individual style. This dissertation presents a comprehensive body of work that introduces a new musical representation that allows users to explore a space of musical rules that are created from their own melodies. This representation, called functional scaffolding for musical composition (FSMC), exploits a simple yet powerful property of multipart compositions: The pattern of notes and rhythms in different instrumental parts of the same song are functionally related. That is, in principle, one part can be expressed as a function of another. Music in FSMC is represented accordingly as a functional relationship between an existing human composition, or scaffold, and an additional generated voice. This relationship is encoded by a type of artificial neural network called a compositional pattern producing network (CPPN). A human user without any musical expertise can then explore how these additional generated voices should relate to the scaffold through an interactive evolutionary process akin to animal breeding. The utility of this insight is validated by two implementations of FSMC called NEAT Drummer and MaestroGenesis, that respectively help users tailor drum patterns and complete multipart arrangements from as little as a single original monophonic track. The five major contributions of this work address the overarching hypothesis in this dissertation that functional relationships alone, rather than specialized music theory, are sufficient for generating plausible additional voices. First, to validate FSMC and determine whether plausible generated voices result from the human-composed scaffold or intrinsic properties of the CPPN, drum patterns are created with NEAT Drummer to accompany several different polyphonic pieces. Extending the FSMC approach to generate pitched voices, the second contribution reinforces the importance of functional transformations through quality assessments that indicate that some partially FSMC-generated pieces are indistinguishable from those that are fully human. While the third contribution focuses on constructing and exploring a space of plausible voices with MaestroGenesis, the fourth presents results from a two-year study where students discuss their creative experience with the program. Finally, the fifth contribution is a plugin for MaestroGenesis called MaestroGenesis Voice (MG-V) that provides users a more natural way to incorporate MaestroGenesis in their creative endeavors by allowing scaffold creation through the human voice. Together, the chapters in this dissertation constitute a comprehensive approach to assisted music generation, enabling creativity without the need for musical expertise

DrawCompileEvolve : sparking interactive evolutionary art with human creations.

This paper presents DrawCompileEvolve, a web-based drawing tool which allows users to draw simple primitive shapes, group them together or define patterns in their groupings (e.g. symmetry, repetition). The user’s vector drawing is then compiled into an indirectly encoded genetic representation, which can be evolved interactively, allowing the user to change the image’s colors, patterns and ultimately transform it. The human artist has direct control while drawing the initial seed of an evolutionary run and indirect control while interactively evolving it, thus making DrawCompileEvolve a mixed-initiative art tool. Early results in this paper show the potential of DrawCompileEvolve to jump-start evolutionary art with meaningful drawings as well as the power of the underlying genetic representation to transform the user’s initial drawing into a different, yet potentially meaningful, artistic rendering.peer-reviewe

Leveraging Human Insights by Combining Multi-Objective Optimization with Interactive Evolution

Deceptive fitness landscapes are a growing concern for evolutionary computation. Recent work has shown that combining human insights with short-term evolution has a synergistic effect that accelerates the discovery of solutions. While humans provide rich insights, they fatigue easily. Previous work reduced the number of human evaluations by evolving a diverse set of candidates via intermittent searches for novelty. While successful at evolving solutions for a deceptive maze domain, this approach lacks the ability to measure what the human evaluator identifies as important. The key insight here is that multi-objective evolutionary algorithms foster diversity, serving as a surrogate for novelty, while measuring user preferences. This approach, called Pareto Optimality-Assisted Interactive Evolutionary Computation (POA-IEC), allows users to identify candidates that they feel are promising. Experimental results reveal that POA-IEC finds solutions in fewer evaluations than previous approaches, and that the non-dominated set is significantly more novel than the dominated set. In this way, POA-IEC simultaneously leverages human insights while quantifying their preferences

Worldwide Infrastructure for Neuroevolution: A Modular Library to Turn Any Evolutionary Domain into an Online Interactive Platform

Across many scientific disciplines, there has emerged an open opportunity to utilize the scale and reach of the Internet to collect scientific contributions from scientists and non-scientists alike. This process, called citizen science, has already shown great promise in the fields of biology and astronomy. Within the fields of artificial life (ALife) and evolutionary computation (EC) experiments in collaborative interactive evolution (CIE) have demonstrated the ability to collect thousands of experimental contributions from hundreds of users across the glob. However, such collaborative evolutionary systems can take nearly a year to build with a small team of researchers. This dissertation introduces a new developer framework enabling researchers to easily build fully persistent online collaborative experiments around almost any evolutionary domain, thereby reducing the time to create such systems to weeks for a single researcher. To add collaborative functionality to any potential domain, this framework, called Worldwide Infrastructure for Neuroevolution (WIN), exploits an important unifying principle among all evolutionary algorithms: regardless of the overall methods and parameters of the evolutionary experiment, every individual created has an explicit parent-child relationship, wherein one individual is considered the direct descendant of another. This principle alone is enough to capture and preserve the relationships and results for a wide variety of evolutionary experiments, while allowing multiple human users to meaningfully contribute. The WIN framework is first validated through two experimental domains, image evolution and a new two-dimensional virtual creature domain, Indirectly Encoded SodaRace (IESoR), that is shown to produce a visually diverse variety of ambulatory creatures. Finally, an Android application built with WIN, filters, allows users to interactively evolve custom image effects to apply to personalized photographs, thereby introducing the first CIE application available for any mobile device. Together, these collaborative experiments and new mobile application establish a comprehensive new platform for evolutionary computation that can change how researchers design and conduct citizen science online

Adapting models of visual aesthetics for personalized content creation

This paper introduces a search-based approach to personalized content generation with respect to visual aesthetics. The approach is based on a two-step adaptation procedure where (1) the evaluation function that characterizes the content is adjusted to match the visual aesthetics of users and (2) the content itself is optimized based on the personalized evaluation function. To test the efficacy of the approach we design fitness functions based on universal properties of visual perception, inspired by psychological and neurobiological research. Using these visual properties we generate aesthetically pleasing 2D game spaceships via neuroevolutionary constrained optimization and evaluate the impact of the designed visual properties on the generated spaceships. The offline generated spaceships are used as the initial population of an interactive evolution experiment in which players are asked to choose spaceships according to their visual taste: the impact of the various visual properties is adjusted based on player preferences and new content is generated online based on the updated computational model of visual aesthetics of the player. Results are presented which show the potential of the approach in generating content which is based on subjective criteria of visual aesthetics.Thanks to all the participants of the interactive evolution experiement. The research was supported, in part, by the FP7 ICT project SIREN (project no: 258453) and by the Danish Research Agency, Ministry of Science, Technology and Innovation project AGameComIn; project number: 274- 09-0083.peer-reviewe

"Knowing is Seeing:" The Digital Audio Workstation and the Visualization of Sound

The computers visual representation of sound has revolutionized the creation of music through the interface of the Digital Audio Workstation software (DAW). With the rise of DAW-based composition in popular music styles, many artists sole experience of musical creation is through the computer screen. I assert that the particular sonic visualizations of the DAW propagate certain assumptions about music, influencing aesthetics and adding new visually-based parameters to the creative process. I believe many of these new parameters are greatly indebted to the visual structures, interactional dictates and standardizations (such as the office metaphor depicted by operating systems such as Apples OS and Microsofts Windows) of the Graphical User Interface (GUI). Whether manipulating text, video or audio, a users interaction with the GUI is usually structured in the same mannerclicking on windows, icons and menus with a mouse-driven cursor. Focussing on the dialogs from the Reddit communities of Making hip-hop and EDM production, DAW user manuals, as well as interface design guidebooks, this dissertation will address the ways these visualizations and methods of working affect the workflow, composition style and musical conceptions of DAW-based producers

Intelligent Tools for Drum Loop Retrieval and Generation

Large libraries of musical data are an increasingly common feature of contemporary computer-based music production practice, with producers often relying heavily on large, curated libraries of data such as loops and samples when making tracks. Drum loop libraries are a particularly common type of library in this context. However, their typically large size, coupled with often poor user interfaces means navigating and exploring them in a fast, easy and enjoyable way is not always possible. Additionally, writing a drum part for a whole track out of many drum loops can be a laborious process, requiring manually editing of many drum loops. The aim of this thesis is to contribute novel techniques based on Music Information Retrieval (MIR) and machine learning that make the process of writing drum tracks using drum loops faster, easier and more enjoyable. We primarily focus on tools for drum loop library navigation and exploration, with additional work on assistive generation of drum loops. We contribute proof-of-concept and prototype tools, Groove Explorer and Groove Explorer 2, for drum loop library exploration based on an interface applying similarity-based visual arrangement of drum loops. Work on Groove Explorer suggested that there were limitations in the existing state-of-the-art approaches to drum loop similarity modelling that must be addressed for tools such as ours to be successful. This was verified via a perceptual study, which identified possible areas of improvement in similarity modelling. Following this, we develop and evaluate a set of novel models for drum loop analysis that capture rhythmic structure and the perceptually relevant qualities of microtiming. Drawing from this, a new approach to drum loop similarity modelling was verified in context as part of Groove Explorer 2, which we evaluated via a user study. The results indicated that our approach could make drum loop library exploration faster, easier and more enjoyable. We finally present an automatic drum loop generation system, jaki, that uses a novel approach for drum loop generation according to user constraints, that could extend Groove Explorer 2 as a drum loop editing and composition tool. Combined, these two systems could offer an end-to-end solution to improved writing of drum tracks

Understanding Collaborative Sensemaking for System Design — An Investigation of Musicians\u27 Practice

There is surprisingly little written in information science and technology literature about the design of tools used to support the collaboration of creators. Understanding collaborative sensemaking through the use of language has been traditionally applied to non-work domains, but this method is also well-suited for informing hypotheses about the design collaborative systems. The presence of ubiquitous, mobile technology, and development of multi-user virtual spaces invites investigation of design which is based on naturalistic, real world, creative group behaviors, including the collaborative work of musicians. This thesis is considering the co-construction of new (musical) knowledge by small groups. Co-construction of new knowledge is critical to the definition of an information system because it emphasizes coordination and resource sharing among group members (versus individual members independently doing their own tasks and only coming together to collate their contributions as a final product). This work situates the locus of creativity on the process itself, rather than on the output (the musical result) or the individuals (members of the band). This thesis describes a way to apply quantitative observations to inform qualitative assessment of the characteristics of collaborative sensemaking in groups. Conversational data were obtained from nine face-to-face collaborative composing sessions, involving three separate bands producing 18 hours of recorded interactions. Topical characteristics of the discussion, namely objects, plans, properties and performance; as well as emergent patterns of generative, evaluative, revision, and management conversational acts within the group were seen as indicative of knowledge construction. The findings report the use of collaborative pathways: iterative cycles of generation, evaluation and revision of temporary solutions used to move the collaboration forward. In addition, bracketing of temporary solutions served to help collaborators reuse content and offload attentional resources. Ambiguity in language, evaluation criteria, goal formation, and group awareness meant that existing knowledge representations were insufficient in making sense of incoming data and necessitated reformulating those representations. Further, strategic use of affective language was found to be instrumental in bridging knowledge gaps. Based on these findings, features of a collaborative system are proposed to help in facilitating sensemaking routines at various stages of a creative task. This research contributes to the theoretical understanding of collaborative sensemaking during non-work, creative activities in order to inform the design of systems for supporting these activities. By studying an environment which forms a potential microcosm of virtual interaction between groups, it provides a framework for understanding and automating collaborative discussion content in terms of the features of dialogue