Usage of Physics Engines for UI Design in NexusUI

In preparation to expand the experimental interfaces in NexusUI widgets, the authors have been evaluating physics engines and exploring physics-based user interfaces on the web. Tying physics simulation events, influenced by user interactions, to web audio encourages exploration of novel methods of interactivity between users and web-based instruments. Object collisions, deformation of a mesh of objects with elastic connections, and liquid simulation via particle generation were identified as systems with dynamics that may provide interesting links to audio synthesis. Two popular physics engines explored are LiquidFun and Matter.js, with new prototype widgets taking advantage of LiquidFun’s Elastic Particles and Matter.js’ Cloth and Newton’s Cradle composites. One of our goals is to discover methods of audio synthesis that complement the behaviors of each physical simulation

Improving User Involvement Through Live Collaborative Creation

Creating an artifact - such as writing a book, developing software, or performing a piece of music - is often limited to those with domain-specific experience or training. As a consequence, effectively involving non-expert end users in such creative processes is challenging. This work explores how computational systems can facilitate collaboration, communication, and participation in the context of involving users in the process of creating artifacts while mitigating the challenges inherent to such processes. In particular, the interactive systems presented in this work support live collaborative creation, in which artifact users collaboratively participate in the artifact creation process with creators in real time. In the systems that I have created, I explored liveness, the extent to which the process of creating artifacts and the state of the artifacts are immediately and continuously perceptible, for applications such as programming, writing, music performance, and UI design. Liveness helps preserve natural expressivity, supports real-time communication, and facilitates participation in the creative process. Live collaboration is beneficial for users and creators alike: making the process of creation visible encourages users to engage in the process and better understand the final artifact. Additionally, creators can receive immediate feedback in a continuous, closed loop with users. Through these interactive systems, non-expert participants help create such artifacts as GUI prototypes, software, and musical performances. This dissertation explores three topics: (1) the challenges inherent to collaborative creation in live settings, and computational tools that address them; (2) methods for reducing the barriers of entry to live collaboration; and (3) approaches to preserving liveness in the creative process, affording creators more expressivity in making artifacts and affording users access to information traditionally only available in real-time processes. In this work, I showed that enabling collaborative, expressive, and live interactions in computational systems allow the broader population to take part in various creative practices.PHDComputer Science & EngineeringUniversity of Michigan, Horace H. Rackham School of Graduate Studieshttps://deepblue.lib.umich.edu/bitstream/2027.42/145810/1/snaglee_1.pd

Exploring Pitch and Timbre through 3D Spaces: Embodied Models in Virtual Reality as a Basis for Performance Systems Design

Our paper builds on an ongoing collaboration between theorists and practitioners within the computer music community, with a specific focus on three-dimensional environments as an incubator for performance systems design. In particular, we are concerned with how to provide accessible means of controlling spatialization and timbral shaping in an integrated manner through the collection of performance data from various modalities from an electric guitar with a multichannel audio output. This paper will focus specifically on the combination of pitch data treated within tonal models and the detection of physical performance gestures using timbral feature extraction algorithms. We discuss how these tracked gestures may be connected to concepts and dynamic relationships from embodied cognition, expanding on performative models for pitch and timbre spaces. Finally, we explore how these ideas support connections between sonic, formal and performative dimensions. This includes instrumental technique detection scenes and mapping strategies aimed at bridging music performance gestures across physical and conceptual planes

Music conducting pedagogy and technology : a document analysis on best practices

This document analysis was designed to investigate pedagogical practices of music conducting teachers in conjunction with research of technologists on the use of various technologies as teaching tools. I sought to discern how conducting teachers and pedagogues are applying recent technological advancements into their teaching strategies. I also sought to understand what paths research is taking about the use of software, hardware, and computer systems applied to the teaching of music conducting technique. This dissertation was guided by four main research questions: (1) How has technology been used to aid in the teaching of conducting? (2) What is the role of technology in the context of conducting pedagogy? (3) Given that conducting is a performative act, how can it be developed through technological means? (4) What technological possibilities exist in the teaching of music conducting technique? Data were collected through music conducting syllabi, conducting textbooks, and research articles. Documents were selected through purposive sampling procedures. Analysis of documents through the constant comparative approach identified emerging themes and differences across the three types of documents. Based on a synthesis of information, I discussed implications for conducting pedagogy and made suggestions for conducting educators.Includes bibliographical references

Proceedings of the 1st International Conference on Live Coding

Open Access peer reviewed papers on live coding published at the 1st International Conference on Live Coding (ICLC) in Leeds

Multimedia Design, Learning Effectiveness, and Student Perceptions of Instructor Credibility and Immediacy

Online learning and the use of multimedia is a quickly growing element of higher education. This experimental research study examines five common audio and video presentation designs to inform evidence-based practices that can be applied by instructional designers as they develop content for online learners. Specifically, this experiment compares instructor-only, slides-only, dual-windows, video-switching, and superimposed-slides multimedia designs in terms of learning effectiveness, perceived instructor credibility, instructor immediacy, and cognitive load created by each design. This study included a diverse sample of adult learners who were randomly assigned to treatment groups. A total of 171 participants completed the study and responded to the NASA Task Load Index (TLX) used to gauge task and cognitive load, the credibility and immediacy survey, and the 20-item post-test. A series of 5x1 Analyses of Variance and Tukey post-hoc calculations were conducted to test for statistically significant differences between groups. The results suggest that a balance can be established between instructor credibility and immediacy by showing both the instructor and instructional content during online classes. The five multimedia designs can yield similar results in recall and comprehension as long as audio, video, and content quality is a design priority. The results also indicate that the design of instructional methods has a greater impact on learning than the device used to receive that instruction. Media and technology are a means to deliver pedagogy and foster communication; it is up to instructors and designers to use evidence-based best practices such as these to build optimal learning environments and instructional systems

Portfolio of Electroacoustic Compositions with Commentaries

This portfolio consists of electroacoustic compositions which were primarily realised through the use of corporeally informed compositional practices. The manner in which a composer interacts with the compositional tools and musical materials at their disposal is a defining factor in the creation of musical works. Although the use of computers in the practice of electroacoustic composition has extended the range of sonic possibilities afforded to composers, it has also had a negative impact on the level of physical interaction that composers have with these musical materials. This thesis is an investigation into the use of mediation technologies with the aim of circumventing issues relating to the physical performance of electroacoustic music. This line of inquiry has led me to experiment with embedded computers, wearable technologies, and a range of various sensors. The specific tools that were used in the creation of the pieces within this portfolio are examined in detail within this thesis. I also provide commentaries and analysis of the eleven electroacoustic works which comprise this portfolio, describing the thought processes that led to their inception, the materials used in their creation, and the tools and techniques that I employed throughout the compositional process

Mobile Music Development Tools for Creative Coders

This project is a body of work that facilitates the creation of musical mobile artworks. The project includes a code toolkit that enhances and simplifies the development of mobile music iOS applications, a flexible notation system designed for mobile musical interactions, and example apps and scored compositions to demonstrate the toolkit and notation system. The code library is designed to simplify the technical aspect of user-centered design and development with a more direct connection between concept and deliverable. This sim- plification addresses learning problems (such as motivation, self-efficacy, and self-perceived understanding) by bridging the gap between idea and functional prototype and improving the ability to contextualize the development process for musicians and other creatives. The toolkit helps to circumvent the need to learn complex iOS development patterns and affords more readable code. CSPD (color, shape, pattern, density) notation is a pseudo-tablature that describes performance interactions. The system leverages visual density and patterns of both color and shape to describe types of gestures (physical or musical) and their relationships rather than focusing on strict rhythmic or pitch/frequency content. The primary design goal is to visualize macro musical concepts that create middleground structure