Conducting a virtual ensemble with a kinect device

This paper presents a gesture-based interaction technique for the implementation of an orchestra conductor and a virtual ensemble, using a 3D camera-based sensor to capture user’s gestures. In particular, a human-computer interface has been developed to recognize conducting gestures using a Microsoft Kinect device. The system allows the conductor to control both the tempo in the piece played as well as the dynamics of each instrument set independently. In order to modify the tempo in the playback, a time-frequency processing-based algorithmis used. Finally, an experiment was conducted to assess user’s opinion of the system as well as experimentally confirm if the features in the system were effectively improving user experience or not.This work has been funded by the Ministerio de Economia y Competitividad of the Spanish Government under Project No. TIN2010-21089-C03-02 and Project No. IPT-2011-0885-430000 and by the Junta de Andalucia under Project No. P11-TIC-7154. The work has been done at Universidad de Malaga. Campus de Excelencia Internacional Andalucia Tech

A quest for the holy grail: Tactile precision, natural movement and haptic feedback in 3D virtual spaces

Three-dimensional immersive spaces such as those provided by virtual worlds, give unparalleled opportunities for learners to practically engage with simulated authentic settings that may be too expensive or too dangerous to experience in the real world. The potential afforded by these environments is severely constrained by the use of a keyboard and mouse moving in two dimensions. While most technologies have evolved rapidly in the early 21st century, the mouse and keyboard as standard navigation and interaction tools have not. However, talented teams from a range of disciplines are on serious quests to address this limitation. Their Holy Grail is to develop ways to interact with 3D immersive spaces using more natural human movements with haptic feedback. Applications would include the training of surgeons and musical conductors, training elite sports people and even physical rehabilitation. This paper reports on the cutting-edge technology projects that look most likely to provide a solution for this complex problem, including the Wiimote and the Microsoft's Project Natal. © 2009 Helen Farley and Caroline Steel

User-based gesture vocabulary for form creation during a product design process

There are inconsistencies between the nature of the conceptual design and the functionalities of the computational systems supporting it, which disrupt the designers’ process, focusing on technology rather than designers’ needs. A need for elicitation of hand gestures appropriate for the requirements of the conceptual design, rather than those arbitrarily chosen or focusing on ease of implementation was identified.The aim of this thesis is to identify natural and intuitive hand gestures for conceptual design, performed by designers (3rd, 4th year product design engineering students and recent graduates) working on their own, without instruction and without limitations imposed by the facilitating technology. This was done via a user centred study including 44 participants. 1785 gestures were collected. Gestures were explored as a sole mean for shape creation and manipulation in virtual 3D space. Gestures were identified, described in writing, sketched, coded based on the taxonomy used, categorised based on hand form and the path travelled and variants identified. Then they were statistically analysed to ascertain agreement rates between the participants, significance of the agreement and the likelihood of number of repetitions for each category occurring by chance. The most frequently used and statistically significant gestures formed the consensus set of vocabulary for conceptual design. The effect of the shape of the manipulated object on the gesture performed, and if the sequence of the gestures participants proposed was different from the established CAD solid modelling practices were also observed.Vocabulary was evaluated by non-designer participants, and the outcomes have shown that the majority of gestures were appropriate and easy to perform. Evaluation was performed theoretically and in the VR environment. Participants selected their preferred gestures for each activity, and a variant of the vocabulary for conceptual design was created as an outcome, that aims to ensure that extensive training is not required, extending the ability to design beyond trained designers only.There are inconsistencies between the nature of the conceptual design and the functionalities of the computational systems supporting it, which disrupt the designers’ process, focusing on technology rather than designers’ needs. A need for elicitation of hand gestures appropriate for the requirements of the conceptual design, rather than those arbitrarily chosen or focusing on ease of implementation was identified.The aim of this thesis is to identify natural and intuitive hand gestures for conceptual design, performed by designers (3rd, 4th year product design engineering students and recent graduates) working on their own, without instruction and without limitations imposed by the facilitating technology. This was done via a user centred study including 44 participants. 1785 gestures were collected. Gestures were explored as a sole mean for shape creation and manipulation in virtual 3D space. Gestures were identified, described in writing, sketched, coded based on the taxonomy used, categorised based on hand form and the path travelled and variants identified. Then they were statistically analysed to ascertain agreement rates between the participants, significance of the agreement and the likelihood of number of repetitions for each category occurring by chance. The most frequently used and statistically significant gestures formed the consensus set of vocabulary for conceptual design. The effect of the shape of the manipulated object on the gesture performed, and if the sequence of the gestures participants proposed was different from the established CAD solid modelling practices were also observed.Vocabulary was evaluated by non-designer participants, and the outcomes have shown that the majority of gestures were appropriate and easy to perform. Evaluation was performed theoretically and in the VR environment. Participants selected their preferred gestures for each activity, and a variant of the vocabulary for conceptual design was created as an outcome, that aims to ensure that extensive training is not required, extending the ability to design beyond trained designers only

The social organisation of the choir rehearsal: How interaction between conductor and choir is used to shape the choir's singing

The aim of this project was to explore the interaction that occurs between conductor and choir in order to develop a better understanding of the ‘unique fingerprint’ of the social activity of choral rehearsals. Little interactional research has previously been carried out in music settings, and what there is mostly focuses on instrumental lessons and masterclasses, which have distinct differences from choirs. The music literature on rehearsals often emphasises best practice, and has a strong focus on school ensembles. Over nineteen hours of choral rehearsal data were collected from eight choirs (nine different conductors; two female), transcribed, and analysed using conversation analysis (CA). The analysis demonstrated many unusual features within the interaction. Findings include a very formal turn-system, with a particularly unusual sung turn in how constrained it is by the conductor’s actions. For example, conductors work hard to launch the turn effectively, may stop it in the midst of the choir’s singing, and use both depiction and verbal utterances to direct, co-construct, and comment on the music while it is ongoing. Directives and assessments are the most prevalent features of the conductors’ feedback turns, and the constant orientation by all parties to improvement over time means that even if only one of the two actions is produced, the other is inferred. The conductors’ feedback also includes large amounts of depiction (including gesture, posture, facial expression and body orientation), verbal description and verbal imagery, which may be used simultaneously to convey more than one meaning or action at the same time. This research contributes to the expanding field of CA research in embodied performance settings, particularly music. It also provides a new methodology for exploring rehearsals in the music literature, which could offer a starting pointing for future research or conductor training programmes

Orchestral conducting since 1950: a comparative analysis of conducting manuals, practitoners' testimonies and two orchestral performances

This dissertation studies the phenomenon of orchestral conducting as it unfolds since 1950 in what may be seen as the international arena of this profession, and does so by comparing three types of texts: respected conducting manuals, leading conductors' testimonies and expert orchestra players' accounts. Two models, empirically inferred from these texts — the Visible Action Continuum and the Thematic String Matrix — are instrumental in subdividing the phenomenon into categories in order to discuss the practitioners' opinions. Scholarly studies then contextualise these discussions. A Video analysis Of Bernstein and Boulez conducting Mahler's Second Symphony complements this text-based approach, aiming to find points of contact between what the practitioners say about orchestral conducting and what the conductors actually do. This video analysis applies the Continuum and the Matrix as well as theories of movement analysis and nonverbal communication. By cross-examining the above- mentioned sources, this study aims to thoroughly discuss the phenomenon Of orchestral conducting. It does not intend to provide direct guidance, theoretical or practical, on how to conduct an orchestra, nor does it propose a standalone score analysis of Mahler's Second Symphony. This study draws the following conclusions: 1. A significant part of the phenomenon of orchestral conducting is not apparent to the observer and exceeds musical boundaries. However both aspects may be accessed through the practitioners' testimonies. 2. The alleged invisibility of some aspects of the phenomenon is better addressed in terms of the observer's unconscious perception. 3. Differences of opinion between practitioners often stem from situational factors: pedagogues, conductors and players see conducting differently. Sometimes sources conflict, but more often they complement each other. 4. The common vocabulary seems unsatisfactory to adequately describe musical praxis and may breed misunderstandings, as much regarding alleged agreements as alleged disagreements between practitioners. 5. As a possible consequence of this insufficiency, only few elements discussed in text-based part of this study are traceable in the conductors' actual performance, and vice versa

ECLAP 2012 Conference on Information Technologies for Performing Arts, Media Access and Entertainment

It has been a long history of Information Technology innovations within the Cultural Heritage areas. The Performing arts has also been enforced with a number of new innovations which unveil a range of synergies and possibilities. Most of the technologies and innovations produced for digital libraries, media entertainment and education can be exploited in the field of performing arts, with adaptation and repurposing. Performing arts offer many interesting challenges and opportunities for research and innovations and exploitation of cutting edge research results from interdisciplinary areas. For these reasons, the ECLAP 2012 can be regarded as a continuation of past conferences such as AXMEDIS and WEDELMUSIC (both pressed by IEEE and FUP). ECLAP is an European Commission project to create a social network and media access service for performing arts institutions in Europe, to create the e-library of performing arts, exploiting innovative solutions coming from the ICT

Artech 2008: proceedings of the 4th International Conference on Digital Arts

ARTECH 2008 is the fourth international conference held in Portugal and Galicia on the topic of Digital Arts. It aims to promote contacts between Iberian and International contributors concerned with the conception, production and dissemination of Digital and Electronic Art. ARTECH brings the scientific, technological and artistic community together, promoting the interest in the digital culture and its intersection with art and technology as an important research field, a common space for discussion, an exchange of experiences, a forum for emerging digital artists and a way of understanding and appreciating new forms of cultural expression. Hosted by the Portuguese Catholic University’s School of Arts (UCP-EA) at the City of Porto, ARTCH 2008 falls in alignment with the main commitment of the Research Center for Science and Technology of the Arts (CITAR) to promote knowledge in the field of the Arts trough research and development within UCP-AE and together with the local and international community. The main areas proposed for the conference were related with sound, image, video, music, multimedia and other new media related topics, in the context of emerging practice of artistic creation. Although non exclusive, the main topics of the conference are usually: Art and Science; Audio-Visual and Multimedia Design; Creativity Theory; Electronic Music; Generative and Algorithmic Art; Interactive Systems for Artistic Applications; Media Art history; Mobile Multimedia; Net Art and Digital Culture; New Experiences with New Media and New Applications; Tangible and Gesture Interfaces; Technology in Art Education; Virtual Reality and Augmented Reality. The contribution from the international community was extremely gratifying, resulting in the submission of 79 original works (Long Papers, Short Papers and installation proposals) from 22 Countries. Our Scientific Committee reviewed these submissions thoroughly resulting in a 73% acceptance ratio of a diverse and promising body of work presented in this book of proceedings. This compilation of articles provides an overview of the state of the art as well as a glimpse of new tendencies in the field of Digital Arts, with special emphasis in the topics: Sound and Music Computing; Technology Mediated Dance; Collaborative Art Performance; Digital Narratives; Media Art and Creativity Theory; Interactive Art; Audiovisual and Multimedia Design.info:eu-repo/semantics/publishedVersio

Computer Music Algorithms. Bio-inspired and Artificial Intelligence Applications

2014 - 2015Music is one of the arts that have most benefited from the invention of computers. Originally, the term Computer Music was used in the scientific community to identify the application of information technology in music composition. It began over time to include the theory and application of new or existing technologies in music, such as sound synthesis, sound design, acoustic, psychoacoustic. Thanks to its interdisciplinary nature, Computer Music can be seen as the encounter of different disciplines. In the last years technology has redefined the way individuals can work, communicate, share experiences, constructively debate, and actively participate to any aspect of the daily life, ranging from business to education, from political and intellectual to social, and also in music activity, such as play music, compose music and so on. In this new context, Computer Music has become an emerging research area for the application of Computational Intelligence techniques, such as machine learning, pattern recognition, bio-inspired algorithms and so on. My research activity is concerned with the Bio-inspired and Artificial Intelligence Applications in the Computer Music. Some of the problems I addressed are summarized in the following. Automatic composition of background music for games, films and other human activities: EvoBackMusic. Systems for real-time composition of background music respond to changes of the environment by generating music that matches the current state of the environment and/or of the user. We propose one such a system that we call EvoBackMusic. It is a multiagent system that exploits a feed-forward neural network and a multi-objective genetic algorithm to produce background music. The neural network is trained to learn the preferences of the user and such preferences are exploited by the genetic algorithm to compose the music. The composition process takes into account a set of controllers that describe several aspects of the environment, like the dynamism of both the user and the 2 context, other physical characteristics, and the emotional state of the user. Previous system mainly focus on the emotional aspect. Publications: • Roberto De Prisco, Delfina Malandrino, Gianluca Zaccagnino, Rocco Zaccagnino: ‘‘An Evolutionary Composer for Real-Time Background Music’’. EvoMUSART 2016: 135-151. Interaction modalities for music performances: MarcoSmiles. In this field we considered new interaction modalities during music performances by using hands without the support of a real musical instrument. Exploiting natural user interfaces (NUI), initially conceived for the game market, it is possible to enhance the traditional modalities of interaction when accessing to technology, build new forms of interactions by transporting users in a virtual dimension, but that fully reflects the reality, and finally, improve the overall perceived experience. The increasing popularity of these innovative interfaces involved their adoption in other fields, including Computer Music. We propose a system, named MarcoSmiles, specifically designed to allow individuals to perform music in an easy, innovative, and personalized way. The idea is to design new interaction modalities during music performances by using hands without the support of a real musical instrument. We exploited Artificial Neural Networks to customize the virtual musical instrument, to provide the information for the mapping of the hands configurations into musical notes and, finally, to train and test these configurations. We performed several tests to study the behavior of the system and its efficacy in terms of learning capabilities. Publications: • Roberto De Prisco, Delfina Malandrino, Gianluca Zaccagnino, Rocco Zaccagnino: ‘‘Natural Users Interfaces to support and enhance Real-Time Music Performance’’. AVI 2016. 3 Bio-inspired approach for automatic music composition Here we describe a new bio-inspired approach for automatic music composition in a specific style: Music Splicing System. Splicing systems were introduced by Tom Head (1987) as a formal model of a recombination process between DNA molecules. The existing literature on splicing systems mainly focuses on the computational power of these systems and on the properties of the generated languages; very few applications based on splicing systems have been introduced. We show a novel application of splicing systems to build an automatic music composer. As a result of a performance study we proved that our composer outperforms other meta-heuristics by producing better music according to a specific measure of quality evaluation, and this proved that the proposed system can be seen also as a new valid bio-inspired strategy for automatic music composition. Publications: ▪ Clelia De Felice, Roberto De Prisco, Delfina Malandrino, Gianluca Zaccagnino, Rocco Zaccagnino, Rosalba Zizza: ‘‘Splicing Music Composition’’. Information Sciences Journal, 385: 196 – 215 (2017). ▪ Clelia De Felice, Roberto De Prisco, Delfina Malandrino, Gianluca Zaccagnino, Rocco Zaccagnino, Rosalba Zizza: ‘‘Chorale Music Splicing System: An Algorithmic Music Composer Inspired by Molecular Splicing’’. EvoMusart 2015: 50 – 61. Music and Visualization Here we describe new approaches for learning of harmonic and melodic rules of classic music, by using visualization techniques: VisualMelody and VisualHarmony. Experienced musicians have the ability to understand the structural elements of music compositions. Such an ability is built over time through the study of music theory, the understanding of rules that guide the composition of music, and through countless hours of practice. The learning process is hard, especially for classical music, where the rigidity of the music structures and styles requires great effort to understand, assimilate, and then master the learned notions. In particular, we focused our attention on a specific type of music compositions, namely, music in chorale style (4-voice music). Composing such type of music 4 is often perceived as a difficult task, because of the rules the composer has to adhere to. In this paper we propose a visualization technique that can help people lacking a strong knowledge of music theory. The technique exploits graphic elements to draw the attention on the possible errors in the composition. We then developed two interactive systems, named VisualMelody and VisualHarmony, that employ the proposed visualization techniques to facilitate the understanding of the structure of music compositions. The aim is to allow people to make 4-voice music composition in a quick and effective way, i.e., avoiding errors, as dictated by classical music theory rules. Publications: ▪ Roberto De Prisco, Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, Rocco Zaccagnino: ‘‘Understanding the structure of music compositions: is visualization an effective approach?’’ Information Visualization Journal, 2016. (DOI): 10.1177/1473871616655468 • Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, Rocco Zaccagnino: ‘‘A Color-Based Visualization Approach to Understand Harmonic Structures of Musical Compositions’’. IV 2015: 56-61. • Delfina Malandrino, Donato Pirozzi, Gianluca Zaccagnino, Rocco Zaccagnino: ‘‘Visual Approaches for Harmonic Analysis of 4-part Music: Implementation and Evaluation’’. Major revision – Journal of Visual Languages and Computing, 2016. [edited by Author]XIV n.s