A Multimodal System for Real-Time Action Instruction in Motor Skill Learning

de Kok I, Hough J, Hülsmann F, Botsch M, Schlangen D, Kopp S. A Multimodal System for Real-Time Action Instruction in Motor Skill Learning. In: Proceedings of ACM ICMI 17th ACM International Conference on Multimodal Interaction. 2015: 355-362

Exposing piaget's scheme: Empirical evidence for the ontogenesis of coordination in learning a mathematical concept

The combination of two methodological resources-natural-user interfaces (NUI) and multimodal learning analytics (MMLA)-is creating opportunities for educational researchers to empirically evaluate seminal models for the hypothetical emergence of concepts from situated sensorimotor activity. 76 participants (9-14 yo) solved tablet-based non-symbolic manipulation tasks designed to foster grounded meanings for the mathematical concept of proportional equivalence. Data gathered in task-based semi-structured clinical interviews included action logging, eye-gaze tracking, and videography. Successful task performance coincided with spontaneous appearance of stable dynamical gaze-path patterns soon followed by multimodal articulation of strategy. Significantly, gaze patterns included uncued non-salient screen locations. We present cumulative results to argue that these 'attentional anchors' mediated participants' problem solving. We interpret the findings as enabling us to revisit, support, refine, and elaborate on central claims of Piaget's theory of genetic epistemology and in particular his insistence on the role of situated motor-action coordination in the process of reflective abstraction

Movement Sonification: Effects on Motor Learning beyond Rhythmic Adjustments

Motor learning is based on motor perception and emergent perceptual-motor representations. A lot of behavioral research is related to single perceptual modalities, but during last two decades the contribution of multimodal perception on motor behavior was discovered more and more. A growing number of studies indicate an enhanced impact of multimodal stimuli on motor perception, motor control and motor learning in terms of better precision and higher reliability of the related actions. Behavioral research is supported by neurophysiological data, revealing that multisensory integration supports motor control and learning. But the overwhelming part of both research lines is dedicated to basic research. Besides research in the domains of music, dance and motor rehabilitation there is nearly no evidence about enhanced effectiveness of multisensory information on learning of gross motor skills. To reduce this gap movement sonification is used here in applied research on motor learning in sports.Based on the current knowledge on the multimodal organization of the perceptual system we generate additional real-time movement information being suitable for integration with perceptual feedback streams of visual and proprioceptive modality. With ongoing training synchronously processed auditory information should be initially integrated into the emerging internal models, enhancing the efficacy of motor learning. This is achieved by a direct mapping of kinematic and dynamic motion parameters to electronic sounds, resulting in continuous auditory and convergent audiovisual or audio-proprioceptive stimulus arrays. In sharp contrast to other approaches using acoustic information as error feedback in motor learning settings we try to generate additional movement information suitable for acceleration and enhancement of adequate sensorimotor representations and processible below the level of consciousness. In the experimental setting participants were asked to learn a closed motor skill (technique acquisition of indoor rowing). One group was treated with visual information and two groups with audiovisual information (sonification vs. natural sounds). For all three groups learning became evident and remained stable. Participants treated with additional movement sonification showed better performance compared to both other groups. Results indicate that movement sonification enhances motor learning of a complex gross motor skill – even exceeding usually expected acoustic rhythmical effects on motor learning

Augmented visual, auditory, haptic, and multimodal feedback in motor learning: A review

It is generally accepted that augmented feedback, provided by a human expert or a technical display, effectively enhances motor learning. However, discussion of the way to most effectively provide augmented feedback has been controversial. Related studies have focused primarily on simple or artificial tasks enhanced by visual feedback. Recently, technical advances have made it possible also to investigate more complex, realistic motor tasks and to implement not only visual, but also auditory, haptic, or multimodal augmented feedback. The aim of this review is to address the potential of augmented unimodal and multimodal feedback in the framework of motor learning theories. The review addresses the reasons for the different impacts of feedback strategies within or between the visual, auditory, and haptic modalities and the challenges that need to be overcome to provide appropriate feedback in these modalities, either in isolation or in combination. Accordingly, the design criteria for successful visual, auditory, haptic, and multimodal feedback are elaborate