Collaborative composition for musical robots

The goal of this research is to collaborate with a number of different artists to explore the capabilities of robotic musical instruments to cultivate new music. This paper describes the challenges faced in using musical robotics in rehearsals and on the performance stage. It also describes the design of custom software frameworks and tools for the variety of composers and performers interacting with the new instruments. Details of how laboratory experiments and rehearsals moved to the concert hall in a variety of diverse performance scenarios are described. Finally, a paradigm for how to teach musical robotics as a multimedia composition course is discussed

Enhancing stroke generation and expressivity in robotic drummers - A generative physics model approach

The goal of this master's thesis research is to enhance the stroke generation capabilities and musical expressivity in robotic drummers. The approach adopted is to understand the physics of human fingers-drumstick-drumhead interaction and try to replicate the same behavior in a robotic drumming system with the minimum number of degrees of freedom. The model that is developed is agnostic to the exact specifications of the robotic drummer that will attempt to emulate human like drum strokes, and therefore can be used in any robotic drummer that uses actuators with complete control over the motor position angle. Initial approaches based on exploiting the instability of a PID control system to generate multiple bounces and the limitations of this approach are also discussed in depth. In order to assess the success of the model and the implementation in the robotic platform a subjective evaluation was conducted. The evaluation results showed that, the observed data was statistically equivalent to the subjects resorting to a blind guess in order to distinguish between a human playing a multiple bounce stroke and a robot playing a similar kind of stroke.M.S

Expressive Musical Robots: Building, Evaluating, and Interfacing with an Ensemble of Mechatronic Instruments

An increase in the number of parameters of expression on musical robots can result in an increase in their expressivity as musical instruments. This thesis focuses on the design, construction, and implementation of four new robotic instruments, each designed to add more parametric control than is typical for the current state of the art of musical robotics. The principles followed in the building of the four new instruments are scalable and can be applied to musical robotics in general: the techniques exhibited in this thesis for the construction and use of musical robotics can be used by composers, musicians, and installation artists to add expressive depth to their own works with robotic instruments. Accompanying the increase in parametric depth applied to the musical robotics is an increase in difficulty in interfacing with them: robots with a greater number of actuators require more time to program. This document aims to address this problem in two ways: the use of closed-loop control for low-level adjustments of the robots and the use of a parametric encoding-equipped musical robot network to provide composers with intuitive musical commands for the robots. The musical robots introduced, described, and applied in this thesis were conceived of as musical instruments for performance and installation use by artists. This thesis closes with an exhibition of the performance and installation uses of these new robots and with a discussion of future research directions

Timbral Learning for Musical Robots

abstract: The tradition of building musical robots and automata is thousands of years old. Despite this rich history, even today musical robots do not play with as much nuance and subtlety as human musicians. In particular, most instruments allow the player to manipulate timbre while playing; if a violinist is told to sustain an E, they will select which string to play it on, how much bow pressure and velocity to use, whether to use the entire bow or only the portion near the tip or the frog, how close to the bridge or fingerboard to contact the string, whether or not to use a mute, and so forth. Each one of these choices affects the resulting timbre, and navigating this timbre space is part of the art of playing the instrument. Nonetheless, this type of timbral nuance has been largely ignored in the design of musical robots. Therefore, this dissertation introduces a suite of techniques that deal with timbral nuance in musical robots. Chapter 1 provides the motivating ideas and introduces Kiki, a robot designed by the author to explore timbral nuance. Chapter 2 provides a long history of musical robots, establishing the under-researched nature of timbral nuance. Chapter 3 is a comprehensive treatment of dynamic timbre production in percussion robots and, using Kiki as a case-study, provides a variety of techniques for designing striking mechanisms that produce a range of timbres similar to those produced by human players. Chapter 4 introduces a machine-learning algorithm for recognizing timbres, so that a robot can transcribe timbres played by a human during live performance. Chapter 5 introduces a technique that allows a robot to learn how to produce isolated instances of particular timbres by listening to a human play an examples of those timbres. The 6th and final chapter introduces a method that allows a robot to learn the musical context of different timbres; this is done in realtime during interactive improvisation between a human and robot, wherein the robot builds a statistical model of which timbres the human plays in which contexts, and uses this to inform its own playing.Dissertation/ThesisDoctoral Dissertation Media Arts and Sciences 201

Designing instruments towards networked music practices

It is commonly noted in New Interfaces for Musical Expression (NIME) research that few of these make it to the mainstream and are adopted by the general public. Some research in Sound and Music Computing (SMC) suggests that the lack of humanistic research guiding technological development may be one of the causes. Many new technologies are invented, however without real aim else than for technical innovation, great products however emphasize the user-friendliness, user involvement in the design process or User-Centred Design (UCD), that seek to guarantee that innovation address real, existing needs among users. Such an approach includes not only traditionally quantifiable usability goals, but also qualitative, psychological, philosophical and musical such. The latter approach has come to be called experience design, while the former is referred to as interaction design. Although the Human Computer Interaction (HCI) community in general has recognized the significance of qualitative needs and experience design, NIME has been slower to adopt this new paradigm. This thesis therefore attempts to investigate its relevance in NIME, and specifically Computer Supported Cooperative Work (CSCW) for music applications by devising a prototype for group music action based on needs defined from pianists engaging in piano duets, one of the more common forms of group creation seen in the western musical tradition. These needs, some which are socio-emotional in nature, are addressed through our prototype although in the context of computers and global networks by allowing for composers from all over the world to submit music to a group concert on a Yamaha Disklavier in location in Porto, Portugal. Although this prototype is not a new gestural controller per se, and therefore not a traditional NIME, but rather a platform that interfaces groups of composers with a remote audience, the aim of this research is on investigating how contextual parameters like venue, audience, joint concert and technologies impact the overall user experience of such a system. The results of this research has been important not only in understanding the processes, services, events or environments in which NIME’s operate, but also understanding reciprocity, creativity, experience design in Networked Music practices.É de conhecimento generalizado que na área de investigação em novos interfaces para expressão musical (NIME - New Interfaces for Musical Expression), poucos dos resultantes dispositivos acabam por ser popularizados e adoptados pelo grande público. Algum do trabalho em computação sonora e musical (SMC- Sound and Music Computing) sugere que uma das causas para esta dificuldade, reside numalacuna ao nível da investigação dos comportamentos humanos como linha orientadora para os desenvolvimentos tecnológicos. Muitos dos desenvolvimentos tecnológicos são conduzidos sem um real objectivo, para além da inovação tecnológica, resultando em excelentes produtos, mas sem qualquer enfâse na usabilidade humana ou envolvimento do utilizador no processo de Design (UCDUser Centered Design), no sentido de garantir que a inovação atende a necessidades reais dos utilizadores finais. Esta estratégia implica, não só objectivos quantitativos tradicionais de usabilidade, mas também princípios qualitativos, fisiológicos, psicológicos e musicológicos. Esta ultima abordagem é atualmente reconhecida como Design de Experiência (Experience Design) enquanto a abordagem tradicional é vulgarmente reconhecida apenas como Design de Interação (Interaction Design). Apesar de na área Interação Homem-Computador (HCI – Human Computer Interaction) as necessidades qualitativas no design de experiência ser amplamente reconhecido em termos do seu significado e aplicabilidade, a comunidade NIME tem sido mais lenta em adoptar este novo paradigma. Neste sentido, esta Tese procura investigar a relevância em NIME, especificamente nu subtópico do trabalho cooperativo suportado por Computadores (CSCW – Computer Supported Cooperative Work), para aplicações musicais, através do desenvolvimento de um protótipo de um sistema que suporta ações musicais coletivas, baseado nas necessidades especificas de Pianistas em duetos de Piano, uma das formas mais comuns de criação musical em grupo popularizada na tradição musical ocidental. Estes requisitos, alguns sócioemocionais na sua natureza, são atendidos através do protótipo, neste caso aplicado ao contexto informático e da rede de comunicações global, permitindo a compositores de todo o mundo submeterem a sua música para um concerto de piano em grupo num piano acústico Yamaha Disklavier, localizado fisicamente na cidade do Porto, Portugal. Este protótipo não introduz um novo controlador em si mesmo, e consequentemente não está alinhado com as típicas propostas de NIME. Trata-se sim, de uma nova plataforma de interface em grupo para compositores com uma audiência remota, enquadrado com objectivos de experimentação e investigação sobre o impacto de diversos parâmetros, tais como o espaço performativo, as audiências, concertos colaborativos e tecnologias em termos do sistema global. O resultado deste processo de investigação foi relevante, não só para compreender os processos, serviços, eventos ou ambiente em que os NIME podem operar, mas também para melhor perceber a reciprocidade, criatividade e design de experiencia nas práticas musicais em rede

Buddy - caixa de ritmos expandida

Este projeto, baptizado com o nome Buddy, trata da construção de uma caixa de ritmos robótica, inspirada e baseada no modelo convencional da caixa de ritmos digital. Concretamente, foi concebida uma bateria musical robótica pela implementação de um sistema de solenóides controlados em tempo real por interfaces MIDI. O robot permite desenvolver o mesmo tipo de trabalho que as caixas de ritmos convencionais possibilitam (e.g. interpretar padrões rítmicos pré-compostos), favorecendo contudo uma experiência sonora acústica e, por conseguinte, uma experiência musical e sonora mais estimulante. Além da abordagem convencional que esta bateria robótica possibilita (e.g. tocar padrões rítmicos), este projecto prevê outras abordagens que possibilitam uma exploração artística e musical mais abrangente, por exemplo, pela implementação de ideias que promovam a interactividade (e.g. controlo da performance do robot pela análise áudio da performance do músico, controlo do robot via internet). O desenvolvimento deste projecto teve como base a plataforma open-source Arduino e foi publicado e documentado online com acesso ao código fonte e aos esquemas electrónicos, permitindo assim que artistas e educadores sem formação em engenharia electrotécnica possam construir robots semelhantes para usar em contextos artísticos ou pedagógicos.This project, named Buddy, deals with the construction of a robotic rhythm box, inspired and based on a conventional model of a digital rhythm box. More specifically, a robotic musical drum set was conceived through the implementation of a real-time automatic system of solenoids controlled via MIDI interfaces. The robot allows the development of the same type of approach that traditional rhythm machines enable (e.g. to interpret pre-composed rhythm patterns), favouring, however, an acoustic sonic experience, and therefore, a more stimulating sound and musical experience. Besides the conventional approach that this robotic drums allows (e.g. to play rhythm patterns), this project foresees other approaches that can enable a wider artistic and musical exploration, for example, by implementing ideas that promote interactivity (e.g. robot performance controled by audio analysis of the musician’s performance, internet robot control). The development of this project was based on the Arduino open-source platform, and it was published and documented online with access to the source code and to the electronic schemes, allowing artists and educators without a formal education in electric and electronics engineering, to be able to build similar robots to use in artistic or pedagogical contexts

Channelisation of Noise through a Rhythmic Grid: Brutalist Mechatronic Sound-sculpture

The aim of this thesis is to provide accessibility and appreciation for sounds that are conventionally perceived as non-musical or “noise”. Ordering the noise on a grid of metric rhythms, and underlining its materiality through an audiovisual mode of expression are the two main strategies employed. Using the medium of mechatronics, mechanically generated sonic by-products of technological developments are chosen as the focus sonic material. As a result, the output of this research extends what is known as glitch music outside the territory of amplified sound, to a realm where noise is created physically and acoustically. Based on these objectives, and following an investigation on the use of mechatronics in contemporary sound-based art, an ensemble of mechatronic sound-sculptures is designed and developed. Varying in terms of material, sound-generating mechanism, and sonic quality, the ensemble is divided into three different instrument-types, each of which is introduced, thoroughly described, and sonically evaluated. Next, three new audiovisual works are developed and realised utilising the mechatronic sound-sculptures, in order to turn into practice the ideas explored in this research. These compositions – which are all exhibited in competitive international symposiums – undertake the integration of mechatronics in three areas of sonic arts that are interconnected with the sound-sculptures. Furthermore, this thesis also establishes an aesthetic framework that formalises a significant body of contemporary sound art and music that, prior to this work, had suffered academic inattention. Probing the various parallels between the ideas developed in this thesis and Brutalist architecture, ‘sound-based brutalism’ is coined and formulated as an aesthetic underpinning for not only the academically marginalised works discussed, but also the work of the author. Lastly, two audiovisual projects (a performance and a series of ten installation pieces) are developed using the entire mechatronic sound-sculpture series in an effort to realise ‘sound-based brutalism’

Adaptive and learning-based formation control of swarm robots

Autonomous aerial and wheeled mobile robots play a major role in tasks such as search and rescue, transportation, monitoring, and inspection. However, these operations are faced with a few open challenges including robust autonomy, and adaptive coordination based on the environment and operating conditions, particularly in swarm robots with limited communication and perception capabilities. Furthermore, the computational complexity increases exponentially with the number of robots in the swarm. This thesis examines two different aspects of the formation control problem. On the one hand, we investigate how formation could be performed by swarm robots with limited communication and perception (e.g., Crazyflie nano quadrotor). On the other hand, we explore human-swarm interaction (HSI) and different shared-control mechanisms between human and swarm robots (e.g., BristleBot) for artistic creation. In particular, we combine bio-inspired (i.e., flocking, foraging) techniques with learning-based control strategies (using artificial neural networks) for adaptive control of multi- robots. We first review how learning-based control and networked dynamical systems can be used to assign distributed and decentralized policies to individual robots such that the desired formation emerges from their collective behavior. We proceed by presenting a novel flocking control for UAV swarm using deep reinforcement learning. We formulate the flocking formation problem as a partially observable Markov decision process (POMDP), and consider a leader-follower configuration, where consensus among all UAVs is used to train a shared control policy, and each UAV performs actions based on the local information it collects. In addition, to avoid collision among UAVs and guarantee flocking and navigation, a reward function is added with the global flocking maintenance, mutual reward, and a collision penalty. We adapt deep deterministic policy gradient (DDPG) with centralized training and decentralized execution to obtain the flocking control policy using actor-critic networks and a global state space matrix. In the context of swarm robotics in arts, we investigate how the formation paradigm can serve as an interaction modality for artists to aesthetically utilize swarms. In particular, we explore particle swarm optimization (PSO) and random walk to control the communication between a team of robots with swarming behavior for musical creation

Sustainable control of infestations using image processing and modelling

A sustainable pest control system integrates automated pest detection and recognition to evaluate the pest density using image samples taken from habitats. Novel predator/prey modelling algorithms assess control requirements for the UAV system, which is designed to deliver measured quantities of naturally beneficial predators to combat pest infestations within economically acceptable timeframes. The integrated system will reduce the damaging effect of pests in an infested habitat to an economically acceptable level without the use of chemical pesticides. Plant pest recognition and detection is vital for food security, quality of life and a stable agricultural economy. The research utilises a combination of the k-means clustering algorithm and the correspondence filter to achieve pest detection and recognition. The detection is achieved by partitioning the data space into Voronoi cells, which tends to find clusters of comparable spatial extents, thereby separating the objects (pests) from the background (pest habitat). The detection is established by extracting the variant and distinctive attributes between the pest and its habitat (leaf, stem) and using the correspondence filter to identify the plant pests to obtain correlation peak values for the different datasets. The correspondence filter can achieve rotationally invariant recognition of pests for a full 360 degrees, which proves the effectiveness of the algorithm and provides a count of the number of pests in the image. A series of models has been produced that will permit an assessment of common pest infestation problems and estimate the number of predators that are required to control the problem within a time schedule. A UAV predator deployment system has been designed. The system is offered as a replacement for chemical pesticides to improve peoples’ health opportunities and the quality of food products