Enhanced visualisation of dance performance from automatically synchronised multimodal recordings

The Huawei/3DLife Grand Challenge Dataset provides multimodal recordings of Salsa dancing, consisting of audiovisual streams along with depth maps and inertial measurements. In this paper, we propose a system for augmented reality-based evaluations of Salsa dancer performances. An essential step for such a system is the automatic temporal synchronisation of the multiple modalities captured from different sensors, for which we propose efficient solutions. Furthermore, we contribute modules for the automatic analysis of dance performances and present an original software application, specifically designed for the evaluation scenario considered, which enables an enhanced dance visualisation experience, through the augmentation of the original media with the results of our automatic analyses

An advanced virtual dance performance evaluator

The ever increasing availability of high speed Internet access has led to a leap in technologies that support real-time realistic interaction between humans in online virtual environments. In the context of this work, we wish to realise the vision of an online dance studio where a dance class is to be provided by an expert dance teacher and to be delivered to online students via the web. In this paper we study some of the technical issues that need to be addressed in this challenging scenario. In particular, we describe an automatic dance analysis tool that would be used to evaluate a student's performance and provide him/her with meaningful feedback to aid improvement

3d augmented mirror: a multimodal interface for string instrument learning and teaching with gesture support

Multimodal interfaces can open up new possibilities for music education, where the traditional model of teaching is based predominantly on verbal feedback. This paper explores the development and use of multimodal interfaces in novel tools to support music practice training. The design of multimodal interfaces for music education presents a challenge in several respects. One is the integration of multimodal technology into the music learning process. The other is the technological development, where we present a solution that aims to support string practice training with visual and auditory feedback. Building on the traditional function of a physical mirror as a teaching aid, we describe the concept and development of an "augmented mirror" using 3D motion capture technology

The Multimodal Tutor: Adaptive Feedback from Multimodal Experiences

This doctoral thesis describes the journey of ideation, prototyping and empirical testing of the Multimodal Tutor, a system designed for providing digital feedback that supports psychomotor skills acquisition using learning and multimodal data capturing. The feedback is given in real-time with machine-driven assessment of the learner's task execution. The predictions are tailored by supervised machine learning models trained with human annotated samples. The main contributions of this thesis are: a literature survey on multimodal data for learning, a conceptual model (the Multimodal Learning Analytics Model), a technological framework (the Multimodal Pipeline), a data annotation tool (the Visual Inspection Tool) and a case study in Cardiopulmonary Resuscitation training (CPR Tutor). The CPR Tutor generates real-time, adaptive feedback using kinematic and myographic data and neural networks

Ubiquitous Integration and Temporal Synchronisation (UbilTS) framework : a solution for building complex multimodal data capture and interactive systems

Contemporary Data Capture and Interactive Systems (DCIS) systems are tied in with various technical complexities such as multimodal data types, diverse hardware and software components, time synchronisation issues and distributed deployment configurations. Building these systems is inherently difficult and requires addressing of these complexities before the intended and purposeful functionalities can be attained. The technical issues are often common and similar among diverse applications. This thesis presents the Ubiquitous Integration and Temporal Synchronisation (UbiITS) framework, a generic solution to address the technical complexities in building DCISs. The proposed solution is an abstract software framework that can be extended and customised to any application requirements. UbiITS includes all fundamental software components, techniques, system level layer abstractions and reference architecture as a collection to enable the systematic construction of complex DCISs. This work details four case studies to showcase the versatility and extensibility of UbiITS framework’s functionalities and demonstrate how it was employed to successfully solve a range of technical requirements. In each case UbiITS operated as the core element of each application. Additionally, these case studies are novel systems by themselves in each of their domains. Longstanding technical issues such as flexibly integrating and interoperating multimodal tools, precise time synchronisation, etc., were resolved in each application by employing UbiITS. The framework enabled establishing a functional system infrastructure in these cases, essentially opening up new lines of research in each discipline where these research approaches would not have been possible without the infrastructure provided by the framework. The thesis further presents a sample implementation of the framework on a device firmware exhibiting its capability to be directly implemented on a hardware platform. Summary metrics are also produced to establish the complexity, reusability, extendibility, implementation and maintainability characteristics of the framework.Engineering and Physical Sciences Research Council (EPSRC) grants - EP/F02553X/1, 114433 and 11394

MIROR-Musical Interaction Relying On Reflexion. Project Final Report

open7siIl progetto è stato valutato dalla Commissione Europea con il massimo del punteggio (15/15) con la seguente valutazione sintetica: Excellent. The proposal successfully addresses all relevant aspects of the criterion in question. Any shortcomings are minor". Ha superato le tre valutazioni annuali con giudizio molto positivo (good) e con brillante giudizio finale. Progetto co-finanziato dalla Comunità Europea, 7° Programma Quadro, ICT-Challenge 4.2, Technology enhanced-learning, Programma Cooperation, no 258338. Il Programma “COOPERATION” è identificato dal DM del 1/7/2011 (Identificazione dei programmi di ricerca di alta qualificazione, finanziati dall'Unione europea o dal Ministero dell'istruzione, dell'universita' e della ricerca di cui all'articolo 29, comma 7, della legge n. 240/2010) come uno dei due programmi di ricerca di alta qualificazione finanziati dall’UE. In dettaglio, dal Coordinatore sono stati ricoperti i seguenti ruoli: • Preparazione della proposta e Coordinameto scientifico del Progetto • Responsabile dei contatti con la Commissione Europea • Supervisione e monitoraggio del lavoro svolto dal Consorzio, attraverso workshops, report tecnici e scientifici e coordinamento dei deliverable • Lieder dei seguenti Work-Packages: • WP1. Project Management • WP5. Psychological Experiments • WP8. Dissemination and Exploitation • Coordinatore dell’ALB • Coordinamento scientifico e organizzativo del gruppo di ricerca dell'Università di Bologna, composto da: o 2 assegni di ricerca post-dottorato o 1 assegno di ricerca o 2 contratti di assistenza alla ricerca o 1 contratto per il sito web o 15 studenti-collaboratori o 3 insegnanti collaboratori • Responsabile delle collaborazioni e convenzioni con l’Istituto Comprensivo di Casalecchio di Reno, la Nuova Scuola di Musica Baroncini di Imola, Il Centro Danza Musikè-Bologna. • Supervisione del Project Management (Dipartimento della Ricerca Europea dell'Università di Bologna - ARIC).The MIROR (Musical Interaction Relying On Reflexion) project is co‐funded by the European Commission under the 7th Framework Programme, Theme ICT‐2009.4.2, Technology‐enhanced learning. MIROR is a three‐years project and started on September 1st, 2010. All information regarding MIROR is available through the MIROR Portal at http://www.mirorproject.eu. The MIROR Project-Final Report deals with the description of the development of an adaptive system for music learning and teaching based on the “reflexive interaction” paradigm. The system is developed in the context of early childhood music education. It acts as an advanced cognitive tutor, designed to promote specific cognitive abilities in the field of music improvisation, both in formal learning contexts (kindergartens, primary schools, music schools) and informal ones (at home, kinder centres, etc.). The reflexive interaction paradigm is based on the idea of letting users manipulate virtual copies of themselves, through specifically designed machine‐learning software referred to as “Interactive Reflexive Musical Systems” (IRMS). By definition IRMS are able to learn and configure themselves according to their understanding of the learner's behaviour. In MIROR the IRMS paradigm is extended with the analysis and synthesis of multisensory expressive gesture to increase its impact on the musical pedagogy of young children, by developing new multimodal interfaces. The project is based on a spiral design approach involving coupled interactions between technical and psychopedagogical partners. MIROR integrates both psychological case‐study experiments, aiming to investigate cognitive hypotheses concerning the mirroring behaviour and the learning efficacy of the platform, and validation studies aiming at developing the software in concrete educational settings. The project contributes to promoting the reflexive interaction paradigm not only in the field of music learning, but more generally as a new paradigm for establishing a synergy between learning and cognition in the context of child/machine interaction.openopenA. R. Addessi ; C. Anagnostopoulou; S. Newman; B. Olsson; F. Pachet; G. Volpe; S. YoungA. R. Addessi ; C. Anagnostopoulou; S. Newman; B. Olsson; F. Pachet; G. Volpe; S. Youn

Landscapes of Affective Interaction: Young Children's Enactive Engagement with Body Metaphors

Empirical research into embodied meaning making suggests specific sensorimotor experiences can support children’s understanding of abstract science ideas. This view is aligned with enactive and grounded cognition perspectives, both centred in the view that our ability to conceptualise emerges from our experiences of interaction with our environment. While much of this research has focused on understanding action and action processes in individual children or children in pairs, less attention has been paid to affective dimensions of young children’s group interaction, and how this relates to meaning making with body metaphors. Indeed, Gallagher describes how no action exists in a vacuum, but rather revolves around a complex web of affective-pragmatic features comprising a ‘Landscape of Interaction’ (2020, p.42). This research project addresses gaps in research in understanding young children’s affective engagement from an enactivist cognition perspective. It takes a Design-Based Research approach with an iterative design orientation to examine young children’s interaction with multisensory body-based metaphors through an embodied participation framework. A series of empirical studies with young children, aged 2-7 years, comprising of experiential workshops, build iteratively upon each other. A novel theoretically informed method, Affective Imagination in Motion, is developed involving several purpose-built multisensory body metaphors prompts to enable access to dimensions of young children’s affective engagement. This research makes theoretical and methodological contributions. It extends the theoretical notion of ‘affect’ from enactive and grounded cognition perspectives through identifying key interactive processes in young children’s engagement with multisensory action metaphors. In addition, the novel method offers a contribution as a way of ‘looking’ at affect within a group situation from affective-pragmatic and social embodiment perspectives. Finally, the research contributes to embodied learning design frameworks offering a guideline for designers wishing to inform their work from enactive cognition perspective