An Experimental Investigation of Modality Effect: Evidence from Eye-Tracking Data

Modality effect is a response that occur when there are manipulations of sensory modality. In this thesis, I present a series of studies about the multimodal processing of visual and auditory presentation. The aim of this dissertation is to investigate how would the different stimulations from different source of modalities affect the oculomotor response. I investigate how different stimuli are processed, recognized and retrieved when they are presented across multiple modalities. Specifically, on question of how would the visual and auditory manipulations influence the oculomotor behaviour. In the research area of the multimodal processing, it has been argued that different kinds of sensory manipulations elicit a distinct kind of cognitive and behavioural response. The study of modality effect is particularly interesting topic for investigations since the world is multimodal in nature. Humans and other living beings are constantly exposed to a wide variety of stimuli rather than to isolated single stimulus. All experiments conducted used an eye-tracking approach since eye-tracking data are known as a reliable measure to study implicit cognitive processing. In Experiment 1, I investigate how different modalities and context interplay on the allocation of visual attention during the perceptual processing of congruent and incongruent multimodal stimuli. In Experiment 2, I investigate recognition memory of multimodal stimuli, focusing on the participants’ reaction to old versus novel stimuli presented in the visual and auditory modalities. In Experiment 3, I monitored looking patterns, to understand how visual and auditory stimuli are mentally reconstructed during mental imagery. I conclude the dissertation with a discussion of how a different kinds of modality manipulations elicit distinct modality effect as revealed by oculomotor response