Sensing and mapping for interactive performance

This paper describes a trans-domain mapping (TDM) framework for translating meaningful activities from one creative domain onto another. The multi-disciplinary framework is designed to facilitate an intuitive and non-intrusive interactive multimedia performance interface that offers the users or performers real-time control of multimedia events using their physical movements. It is intended to be a highly dynamic real-time performance tool, sensing and tracking activities and changes, in order to provide interactive multimedia performances. From a straightforward definition of the TDM framework, this paper reports several implementations and multi-disciplinary collaborative projects using the proposed framework, including a motion and colour-sensitive system, a sensor-based system for triggering musical events, and a distributed multimedia server for audio mapping of a real-time face tracker, and discusses different aspects of mapping strategies in their context. Plausible future directions, developments and exploration with the proposed framework, including stage augmenta tion, virtual and augmented reality, which involve sensing and mapping of physical and non-physical changes onto multimedia control events, are discussed

Discovering Design Possibilities through a Pedagogy of Multiliteracies

Research and educational policies have alerted teachers to the importance of multiliteracies. Communication in society today is characterised by rapidly changing and emergent forms of meaning-making in a context of increased cultural and linguistic diversity. This paper responds to these imperatives, releasing key findings of a critical ethnography concerning interactions between pedagogy and access to multiliteracies among culturally and linguistically diverse learners. Data collection involved 18 days of lesson observations over 10 weeks using field and journal notes, continuous audiovisual and audio recording, and the collection of cultural artefacts. Semi-structured interviewing was also conducted with the teacher, principal, and four students. Data analytic tools included low and high inference coding and pragmatic horizon analysis. Findings concerned the use of overt instruction and situated practice in the teacher’s enactment of the multiliteracies pedagogy. This had a significant influence on the learners’ability to access claymation movie designing. Comparisons are made between the learning that occurred for students of the dominant, Anglo-Australian, middle-class culture, and for those who were not. The conclusion addresses relevant literature concerning how to apply the multiliteracies pedagogy to enable meaningful designing

Pedagogies of Design and Multiliterate Learner Identities

In an era of multiliteracies, teaching and learning have become knowledge performances at multiple levels. Instead of a singular, linear focus upon print technologies, the techno-oriented philosophy of teaching aims at providing a rhizomatic network of texts where there is a close link between, and often an overlap of, different designs—linguistic, visual, spatial, and gestural—to construct the multiliterate learner. In this paper, I discuss the role of multimodal literacies in a primary classroom, affirming the role of multiliteracies and decentring the pre-dominance of linguistic at the cost of other designs. While the print media are acknowledged as significant to literacy, the multimodality of print is enhanced through visual and spatial design (Kenner, 2004). Through graphic examples of ICT applications of designs in a primary classroom, I demonstrate that students are operating through multitextual and digitextual (Everett, 2003) practices. What follows is the complex positioning and re-situating of teacher and learner identities engaged in learning through the knowledge processes of experiencing, identifying, applying and critiquing concepts (Kalantzis & Cope, 2004). In particular, I argue that within the diversity of present day classrooms, the digital oriented, multiliterate learner is implicated in constant identity construction by drawing upon macro and micro social practices. I conclude by reiterating the significance of new technologies and new literacy practices as essential to the construction of new learner identities

Interactive Spaces. Models and Algorithms for Reality-based Music Applications

Reality-based interfaces have the property of linking the user's physical space with the computer digital content, bringing in intuition, plasticity and expressiveness. Moreover, applications designed upon motion and gesture tracking technologies involve a lot of psychological features, like space cognition and implicit knowledge. All these elements are the background of three presented music applications, employing the characteristics of three different interactive spaces: a user centered three dimensional space, a floor bi-dimensional camera space, and a small sensor centered three dimensional space. The basic idea is to deploy the application's spatial properties in order to convey some musical knowledge, allowing the users to act inside the designed space and to learn through it in an enactive way

Sketching sonic interactions by imitation-driven sound synthesis

Sketching is at the core of every design activity. In visual design, pencil and paper are the preferred tools to produce sketches for their simplicity and immediacy. Analogue tools for sonic sketching do not exist yet, although voice and gesture are embodied abilities commonly exploited to communicate sound concepts. The EU project SkAT-VG aims to support vocal sketching with computeraided technologies that can be easily accessed, understood and controlled through vocal and gestural imitations. This imitation-driven sound synthesis approach is meant to overcome the ephemerality and timbral limitations of human voice and gesture, allowing to produce more refined sonic sketches and to think about sound in a more designerly way. This paper presents two main outcomes of the project: The Sound Design Toolkit, a palette of basic sound synthesis models grounded on ecological perception and physical description of sound-producing phenomena, and SkAT-Studio, a visual framework based on sound design workflows organized in stages of input, analysis, mapping, synthesis, and output. The integration of these two software packages provides an environment in which sound designers can go from concepts, through exploration and mocking-up, to prototyping in sonic interaction design, taking advantage of all the possibilities of- fered by vocal and gestural imitations in every step of the process