Conducting a virtual ensemble with a kinect device

This paper presents a gesture-based interaction technique for the implementation of an orchestra conductor and a virtual ensemble, using a 3D camera-based sensor to capture user’s gestures. In particular, a human-computer interface has been developed to recognize conducting gestures using a Microsoft Kinect device. The system allows the conductor to control both the tempo in the piece played as well as the dynamics of each instrument set independently. In order to modify the tempo in the playback, a time-frequency processing-based algorithmis used. Finally, an experiment was conducted to assess user’s opinion of the system as well as experimentally confirm if the features in the system were effectively improving user experience or not.This work has been funded by the Ministerio de Economia y Competitividad of the Spanish Government under Project No. TIN2010-21089-C03-02 and Project No. IPT-2011-0885-430000 and by the Junta de Andalucia under Project No. P11-TIC-7154. The work has been done at Universidad de Malaga. Campus de Excelencia Internacional Andalucia Tech

A Conceptual Framework for Motion Based Music Applications

Imaginary projections are the core of the framework for motion based music applications presented in this paper. Their design depends on the space covered by the motion tracking device, but also on the musical feature involved in the application. They can be considered a very powerful tool because they allow not only to project in the virtual environment the image of a traditional acoustic instrument, but also to express any spatially defined abstract concept. The system pipeline starts from the musical content and, through a geometrical interpretation, arrives to its projection in the physical space. Three case studies involving different motion tracking devices and different musical concepts will be analyzed. The three examined applications have been programmed and already tested by the authors. They aim respectively at musical expressive interaction (Disembodied Voices), tonal music knowledge (Harmonic Walk) and XX century music composition (Hand Composer)

MotionDesigner: Augmented artistic performances with kinect-based human body motion tracking

In the last two decades the use of technology in art projects has proliferated, as is the case of the interactive projections based on movement used in artistic performances and installations. However, the artists responsible for creating this work typically have to rely on computer experts to implement this type of interactive systems. The tool herein presented, MotionDesigner, intends to assist the design of these systems by providing artists with higher levels of autonomy and efficiency during the creative process, allowing them to specify the rules by which a human body interacts with both the audio and the visuals used in their interactive art work. The presented tool relies on an RGB-D camera to modulate the multimedia content according to the performer’s body motion. MotionDesigner is extensible so as to accommodate additions required by artists. The tool has been tested with dancers, choreographers, and architects. Results show that MotionDesigner is a valuable aid to artists, working as a catalyst of their creative process.info:eu-repo/semantics/acceptedVersio

Gesture, sound and place.

This paper discusses two multichannel interactive audiovisual artworks, Action A/V and SoundLabyrinth, that explore approaches to the experience of gesture, sound and place. Both works were situated in a geodesic dome frame and built within the Max, Ableton and Max for Live computer programs, and produced ostensibly similar outcomes, however the approaches taken by the two authors differ in intention, processes, and philosophy. These approaches were presented and discussed in workshops delivered at ISEA2013, on Sunday June 9 and Monday June 10 2013. In these workshops participants mprovised with the two systems, both through moving in the dome and by operating the related software, and discussed approaches and understandings of the three terms listed in the title of this paper

Virtual Conductor for String Quartet Practice

This paper presents a system that emulates an ensemble conductor for string quartets. This application has been developed as a support tool for individual and group practice, so that users of any age range can use it to further hone their skills, both for regular musicians and students alike. The virtual conductor designed can offer similar indications to those given by a real ensemble conductor to potential users regarding beat times, dynamics, etc. The application developed allows the user to rehearse his/her performance without the need of having an actual conductor present, and also gives access to additional tools to further support the learning/practice process, such as a tuner or a melody evaluator. The system developed also allows for both solo practice and group practice. A set of tests were conducted to check the usefulness of the application as a practice support tool. A group of musicians from the Chamber Orchestra of Malaga including an ensemble conductor tested the system, and reported to have found it a very useful tool within an educational environment and that it helps to address the lack of this kind of educational tools in a self-learning environment.This work has been funded by the Ministerio de Economia y Competitividad of the Spanish Government under Project No. TIN2010-21089-C03- 02 and Project No. IPT-2011-0885-430000 and by the Ministerio de Industria, Turismo y Comercio under Project No. TSI-090100-2011-25

Towards the Design of a Natural User Interface for Performing and Learning Musical Gestures

AbstractA large variety of musical instruments, either acoustical or digital, are based on a keyboard scheme. Keyboard instruments can produce sounds through acoustic means but they are increasingly used to control digital sound synthesis processes with nowadays music. Interestingly, with all the different possibilities of sonic outcomes, the input remains a musical gesture. In this paper we present the conceptualization of a Natural User Interface (NUI), named the Intangible Musical Instrument (IMI), aiming to support both learning of expert musical gestures and performing music as a unified user experience. The IMI is designed to recognize metaphors of pianistic gestures, focusing on subtle uses of fingers and upper-body. Based on a typology of musical gestures, a gesture vocabulary has been created, hierarchized from basic to complex. These piano-like gestures are finally recognized and transformed into sounds

RGB-D datasets using microsoft kinect or similar sensors: a survey

RGB-D data has turned out to be a very useful representation of an indoor scene for solving fundamental computer vision problems. It takes the advantages of the color image that provides appearance information of an object and also the depth image that is immune to the variations in color, illumination, rotation angle and scale. With the invention of the low-cost Microsoft Kinect sensor, which was initially used for gaming and later became a popular device for computer vision, high quality RGB-D data can be acquired easily. In recent years, more and more RGB-D image/video datasets dedicated to various applications have become available, which are of great importance to benchmark the state-of-the-art. In this paper, we systematically survey popular RGB-D datasets for different applications including object recognition, scene classification, hand gesture recognition, 3D-simultaneous localization and mapping, and pose estimation. We provide the insights into the characteristics of each important dataset, and compare the popularity and the difficulty of those datasets. Overall, the main goal of this survey is to give a comprehensive description about the available RGB-D datasets and thus to guide researchers in the selection of suitable datasets for evaluating their algorithms

Motion Based Learning for Preschools Mathematics via Kinect (Counting Number)

Kids start learning numbers since they in preschools. Every kids have difficulties to start learn about mathematics. They do not recognize and might be difficult for them to remember the numbers at first. Teacher will help them to teach and assist them to solve this problem and it usually happens after some stages and time. The only problem is education system use currently is not helping much in increasing the speed of children's learning. With current education system that sometime bored and only use whiteboard as medium of teaching, kids cannot give their full commitment in class. In Malaysia, traditional and conventional teaching still widely being practiced. However, with the advancement of technology nowadays, it can help to overcome this problem. By using current technology, it can help to complement current education system. Education system can utilize the advancement of technology to overcome problem such as boring and not interactive learning. Example of technology that can be use is Kinect. Kinect is a device that can track human body to interact with an application. It can help make learning become active, interactive and at it also involving all body parts. The objective of introducing kinect in education is to develop a motion based learning application for preschools' mathematics. To achieve this objective, the author has decided to test the application for preschools kid age from 3 to 6 years old. It will focus on teaching simple numbers. Kinect in mathematic will cover on basic numbers among preschool kids. Since the time given to develop this project is very limited, the methodology chosen for this development is throwaway prototype methodology. By using throwaway prototyping methodology this project have improved and increase user involvement. Results showing that kids enjoy the kinect technology in mathematics and teachers have found one of interactive ways of teaching. From data collected and analyzed, results showed that kids enjoy this kinect technology in mathematics and at same time teachers have found one of interactive ways of teaching