An Animation Framework for Continuous Interaction with Reactive Virtual Humans

We present a complete framework for animation of Reactive Virtual Humans that offers a mixed animation paradigm: control of different body parts switches between keyframe animation, procedural animation and physical simulation, depending on the requirements of the moment. This framework implements novel techniques to support real-time continuous interaction. It is demonstrated on our interactive Virtual Conductor

When I count to four…: James Brown, Kraftwerk, and the practice of musical time-keeping before Techno

Of all creative artists, wrote Hector Berlioz in his famous orchestration treatise, the composer is almost the only one to depend on a host of intermediaries between him and his audience (Berlioz, 2002 [1856]: 336). These intermediaries – the orchestra and its leader and time keeper, the conductor – may be intelligent or stupid, devoted or hostile, energetic or lazy; from first to last they can contribute to the glory of [the] work, or they can spoil it, insult it, or even wreck it completely (Ibid.). From written score to performance, realizing a composer\u27s work of music becomes an acute problem of both collective action and aesthetic interpretation. The chief mediator between the composer\u27s artistic intention and its social realization is the conductor, who through his or her authority not only asserts and determines the tempo of a performance, but also establishes its nuance, feeling, and overall interpretation

Leading and following with a virtual trainer

This paper describes experiments with a virtual fitness trainer capable of mutually coordinated interaction. The virtual human co-exercises along with the user, leading as well as following in tempo, to motivate the user and to influence the speed with which the user performs the exercises. In a series of three experiments (20 participants in total) we attempted to influence the users' performance by manipulating the (timing of the) exercise behavior of the virtual trainer. The results show that it is possible to do this implicitly, using only micro adjustments to its bodily behavior. As such, the system is a rst step in the direction of mutually coordinated bodily interaction for virtual humans

Music conducting pedagogy and technology : a document analysis on best practices

This document analysis was designed to investigate pedagogical practices of music conducting teachers in conjunction with research of technologists on the use of various technologies as teaching tools. I sought to discern how conducting teachers and pedagogues are applying recent technological advancements into their teaching strategies. I also sought to understand what paths research is taking about the use of software, hardware, and computer systems applied to the teaching of music conducting technique. This dissertation was guided by four main research questions: (1) How has technology been used to aid in the teaching of conducting? (2) What is the role of technology in the context of conducting pedagogy? (3) Given that conducting is a performative act, how can it be developed through technological means? (4) What technological possibilities exist in the teaching of music conducting technique? Data were collected through music conducting syllabi, conducting textbooks, and research articles. Documents were selected through purposive sampling procedures. Analysis of documents through the constant comparative approach identified emerging themes and differences across the three types of documents. Based on a synthesis of information, I discussed implications for conducting pedagogy and made suggestions for conducting educators.Includes bibliographical references

A Demonstration of Continuous Interaction with Elckerlyc

We discuss behavior planning in the style of the SAIBA framework for continuous (as opposed to turn-based) interaction. Such interaction requires the real-time application of minor shape or timing modifications of running behavior and anticipation of behavior of a (human) interaction partner. We discuss how behavior (re)planning and on-the-fly parameter modification fit into the current SAIBA framework, and what type of language or architecture extensions might be necessary. Our BML realizer Elckerlyc provides flexible mechanisms for both the specification and the execution of modifications to running behavior. We show how these mechanisms are used in a virtual trainer and two turn taking scenarios

Cognitive Robots for Social Interactions

One of my goals is to work towards developing Cognitive Robots, especially with regard to improving the functionalities that facilitate the interaction with human beings and their surrounding objects. Any cognitive system designated for serving human beings must be capable of processing the social signals and eventually enable efficient prediction and planning of appropriate responses. My main focus during my PhD study is to bridge the gap between the motoric space and the visual space. The discovery of the mirror neurons ([RC04]) shows that the visual perception of human motion (visual space) is directly associated to the motor control of the human body (motor space). This discovery poses a large number of challenges in different fields such as computer vision, robotics and neuroscience. One of the fundamental challenges is the understanding of the mapping between 2D visual space and 3D motoric control, and further developing building blocks (primitives) of human motion in the visual space as well as in the motor space. First, I present my study on the visual-motoric mapping of human actions. This study aims at mapping human actions in 2D videos to 3D skeletal representation. Second, I present an automatic algorithm to decompose motion capture (MoCap) sequences into synergies along with the times at which they are executed (or "activated") for each joint. Third, I proposed to use the Granger Causality as a tool to study the coordinated actions performed by at least two units. Recent scientific studies suggest that the above "action mirroring circuit" might be tuned to action coordination rather than single action mirroring. Fourth, I present the extraction of key poses in visual space. These key poses facilitate the further study of the "action mirroring circuit". I conclude the dissertation by describing the future of cognitive robotics study

Violins, venues and vortexes : interrogating pre-reflective relationality in orchestral work : a thesis presented in partial fulfilment of the requirements for the degree of Doctor of Philosophy in Management at Massey University, Manawatū Campus, Palmerston North, New Zealand

This thesis explores the social structures of organising through an analysis of pre-reflective relationality in orchestral performance across three exemplary settings. These are: the opening stanza of a performance by the orchestra in which I play; a highly regarded performance by a well-known orchestra and conductor; and a concert performed under the shadow of COVID-19. Within these contexts, the player’s relationship with instrument and score, the role of the conductor, relations between conductor and player, and the player’s relations with audience, artifact and colleague are discussed. The study draws on autoethnography and the descriptive phenomenological method of Giorgi (2012). This framework allows work practices that are specialized, tacit, and entrenched to be interrogated through the theoretical lens of Merleau-Ponty’s (1968) late ontology as represented by the constructs of reversibility, écart, and Flesh. The research contributes to organisational knowledge on three dimensions. The contribution to theory is made through the interrogation of the pre-reflective relational bonds in symphony orchestras, first between individuals and artifacts, and then between individuals and colleagues, which shape the inter-collegial ‘between space’ (Ladkin, 2013) where the organizing of performance – the music-making itself – happens. The contribution to method is made in the exploration of specialized personal experience for research purposes through Giorgi’s framework and Merleau-Ponty’s constructs, while the contribution to practice builds on this foundation by using Merleau-Ponty’s ideas to acknowledge the inanimate alongside the human and so offer a fresh starting point for the understanding of organizational relationality. This approach also allows orchestral performance to emerge as a primordially interwoven, inherently reversible meshwork of relational connectivity harnessed in pursuit of a collective purpose. As organizations look beyond COVID-19 to a world where the virtual and hybrid must be accommodated alongside the longstanding and traditional, holistic approaches such as the one offered here will resonate with researchers and managers alike as they come to terms with relational structures and organizational contexts transformed by the combined effects of pandemic-related disruption and technological change

A Vision-based Real-time Conductor Gesture Tracking System

[[abstract]]In recent years, interaction between humans and computers is becoming more important. “Virtual Orchestra” is an Human Computer Interface (HCI) software which attempts to authentically reproduce a live orchestra using synthesized and sampled instruments sounds. Compared with the traditional HCIs, using vision-based gesture can provide a touch-free interface which is less bounding than mechanical instruments. In this research, we design a vision-based system that can track the hand motions of a conductor from webcam and extract musical beats from motions. The algorithm used is based on a robust nonparametric technique for climbing density gradients to find the mode of probability distributions. For each frame, the mean shift algorithm converges to the mode of the distribution. Then, the CAMSHIFT algorithm is used to track the moving objects in a video scene. After acquiring the target center point continuously, we can form the trajectory of moving target (such as baton, conductor’s hand…etc). By computing an approximation of k-curvature for the trajectory, and the angle between these two motion vectors, we can compute the point of the change of direction. In this thesis, a system was developed for interpreting a conductor’s gestures and translating theses gestures into musical beats that can be explained as the major part of the music. This system does not require the use of active sensing, special baton, or other constraints on the physical motion of the conductor.