Don’t Forget the Laptop: Using Native Input Capabilities for Expressive Musical Control

We draw on our experiences with the Princeton Laptop Orchestra to discuss novel uses of the laptop’s native physical inputs for flexible and expressive control. We argue that instruments designed using these built-in inputs offer benefits over custom standalone controllers, particularly in certain group performance settings; creatively thinking about native capabilities can lead to interesting and unique new interfaces. We discuss a variety of example instruments that use the laptop’s native capabilities and suggest avenues for future work. We also describe a new toolkit for rapidly experimenting with these capabilities

BlockyTalky: A Physical and Distributed Computer Music Toolkit for Kids

NIME research realizes a vision of performance by means of computational expression, linking body and space to sound and imagery through eclectic forms of sensing and interaction. This vision could dramatically impact computer science education, simultaneously modernizing the field and drawing in diverse new participants. We describe our work creating a NIME-inspired computer music toolkit for kids called BlockyTalky; the toolkit enables users to create networks of sensing devices and synthesizers. We offer findings from our research on student learning through programming and performance. We conclude by suggesting a number of future directions for NIME researchers interested in education

Using Distributed Cognition Theory to Analyze Collaborative Computer Science Learning

Research on students’ learning in computing typically investigates how to enable individuals to develop concepts and skills, yet many forms of computing education, from peer instruction to robotics competitions, involve group work in which understanding may not be entirely locatable within individuals’ minds. We need theories and methods that allow us to understand learning in cognitive systems: culturally and historically situated groups of students, teachers, and tools. Accordingly, we draw on Hutchins’ Distributed Cognition [16] theory to present a qualitative case study analysis of interaction and learning within a small group of middle school students programming computer music. Our analysis shows how a system of students, teachers, and tools, working in a music classroom, is able to accomplish conceptually demanding computer music programming. We show how the system does this by 1) collectively drawing on individuals’ knowledge, 2) using the physical and virtual affordances of different tools to organize work, externalize knowledge, and create new demands for problem solving, and 3) reconfiguring relationships between individuals and tools over time as the focus of problem solving changes. We discuss the implications of this perspective for research on teaching, learning and assessment in computing

Tangible Distributed Computer Music for Youth

Computer music research realizes a vision of performance by means of computational expression, linking body and space to sound and imagery through eclectic forms of sensing and interaction. This vision could dramatically impact computer science education, simultaneously modernizing the field and drawing in diverse new participants. In this article, we describe our work creating an interactive computer music toolkit for kids called BlockyTalky. This toolkit enables users to create networks of sensing devices and synthesizers, and to program the musical and interactive behaviors of these devices. We also describe our work with two middle school teachers to co-design and deploy a curriculum for 11- to 13-year-old students. We draw on work with these students to evidence how computer music can support learning about computer science concepts and change students’ perceptions of computing. We conclude by outlining some remaining questions around how computer music and computer science may best be linked to provide transformative educational experiences

Mapping Through Listening

Gesture-to-sound mapping is generally defined as the association between gestural and sound parameters. This article describes an approach that brings forward the perception-action loop as a fundamental design principle for gesture–sound mapping in digital music instrument. Our approach considers the processes of listening as the foundation – and the first step – in the design of action-sound relationships. In this design process, the relationship between action and sound is derived from actions that can be perceived in the sound. Building on previous works on listening modes and gestural descriptions we proposed to distinguish between three mapping strategies: instantaneous, temporal, and metaphoric. Our approach makes use of machine learning techniques for building prototypes, from digital music instruments to interactive installations. Four different examples of scenarios and prototypes are described and discussed

Including Smart Architecture in environments for people with dementia

Environments which aim to promote human well-being must address both functional and psychosocial needs. This paper comprises a description of a framework for a smart home environment, which aims to comprehensively address issues of environmental fit, in particular for a person with cognitive impairment associated with dementia, by means of introducing sensing of user affect as a factor in system management of a smart personal life space, and in generation of environmental response, adapting to changing user need. The introduction of affective computing into an intelligent system managing environmental response and adaptation is seen as a critical component in successfully realizing an interactive personal life-space, where a continuous feedback loop operates between user and environment, in real time. The overall intention is to maximize environmental congruence for the user, both functionally and psychosocially, by factoring in adjustment to changing user status. Design thinking, at all scales, is perceived as being essential to achieving a coherent smart environment, where architecture is reframed as interaction design

Human model evaluation in interactive supervised learning

Model evaluation plays a special role in interactive machine learning (IML) systems in which users rely on their assessment of a model's performance in order to determine how to improve it. A better understanding of what model criteria are important to users can therefore inform the design of user interfaces for model evaluation as well as the choice and design of learning algorithms. We present work studying the evaluation practices of end users interactively building supervised learning systems for real-world gesture analysis problems. We examine users' model evaluation criteria, which span conventionally relevant criteria such as accuracy and cost, as well as novel criteria such as unexpectedness. We observed that users employed evaluation techniques---including cross-validation and direct, real-time evaluation---not only to make relevant judgments of algorithms' performance and interactively improve the trained models, but also to learn to provide more effective training data. Furthermore, we observed that evaluation taught users about what types of models were easy or possible to build, and users sometimes used this information to modify the learning problem definition or their plans for using the trained models in practice. We discuss the implications of these findings with regard to the role of generalization accuracy in IML, the design of new algorithms and interfaces, and the scope of potential benefits of incorporating human interaction in the design of supervised learning systems