Exploring crossmodal perceptual enhancement and integration in a sequence-reproducing task with cognitive priming

Leveraging the perceptual phenomenon of crossmoal correspondence has been shown to facilitate peoples information processing and improves sensorimotor performance. However for goal-oriented interactive tasks, the question of how to enhance the perception of specific Crossmodal information, and how Crossmodal information integration takes place during interaction is still unclear. The present paper reports two experiments investigating these questions. In the first experiment, a cognitive priming technique was introduced as a way to enhance the perception of two Crossmodal stimuli, in two conditions respectively, and their effect on sensory-motor performance was observed. Based on the results, the second experiment combined the two Crossmodal stimuli in the same interfaces in a way that their correspondence congruency was mutually exclusive. The same priming techniques was applied as a manipulating factor to observe the Crossmodal integration process. Results showed that first, the Crossmodal integration during interaction can be enhanced by the priming technique, but the effect varies according to the combination of Crossmodal stimuli and the types of priming material. Moreover, peoples subjective evaluations towards priming types were in contradiction with their objective behavioural data. Second, when two Crossmodal sequences can be perceived simultaneously, results suggested different perceptual weights are possessed by different participants, and the perceptual enhancement effect was observed only on the dominant one, the pitch-elevation. Furthermore, the Crossmodal integration tended to be integrated in a selective manner without priming. These results contribute design implications for multisensory feedback and mindless computing

Can rhythm be touched? An evaluation of rhythmic sketch performance with augmented multimodal feedback

Although it has been shown that augmented multimodal feedback has a facilitatory effect on motor performance for motor learning and music training, the functionality of haptic feedback combined with other modalities in rhythmic movement tasks has rarely been explored and analysed. In this paper, we evaluate the functionality of visual-haptic feedback in a rhythmic sketch task by comparing it with other multimodal conditions. Further, we examine the possibility of accessing the quality of task execution through kinematic analysis. Based on participants' speed profiles, we investigate the quality of motor control and movement smoothness under different feedback conditions. Results revealed better motor control ability with auditory feedback and improved movement smoothness with haptic feedback. Finally, we propose that haptic feedback can be integrated with other modal stimuli for different interaction purposes, and that kinematic analysis can be a complementary approach to gesture analysis as well as providing subjective evaluation of interaction performance