Insights into the Interconnected Brain during (Multi-)sensory Processing - A concurrent TMS-fMRI Approach

The ability to appropriately respond to sensory information from our surroundings relies on the dynamic interplay between different and distributed brain regions, which is flexibly adapted according to current contexts and demands. By allowing direct monitoring of local and distal effects of transcranial magnetic stimulation (TMS), the concurrent application of TMS and functional magnetic resonance imaging (fMRI) provides a causal interventional approach to investigate this interconnected nature of the brain. In this dissertation, we used this methodology to investigate the neural mechanisms underlying (audio-)visual processing under different cognitive and experimental settings. In particular, given the functional heterogeneity of the intraparietal sulcus (IPS) in a number of different cognitive functions, a special focus was given to the functional role of this region during such processes. As a first step, we evaluated the causal involvement of the IPS during crossmodal deactivations by applying continuous repetitive TMS at different intensity levels to the right IPS and at the Vertex during three different sensory contexts (visual, auditory and fixation). Second, by engaging the attentional network in a demanding visual detection task we investigated how TMS at the right IPS influenced task-related activations, by applying bursts of TMS pulses over the right IPS and during a Sham condition. Moreover, given that additional sensory information might influence task performance in a beneficial or in a detrimental way, we further manipulated the bottom-up sensory context by introducing two different auditory contexts (present vs. absent). Third, to evaluate the differential effects of stimulating low-level sensory areas and higher-order association cortices, we compared the consequences of parietal and occipital stimulation on task-related activations during an identical experimental setting. Lastly, keeping the same visual detection task, we assessed the role of the IPS during perceptual decisions by categorizing participants’ responses into hits, misses, false alarms and correct rejections and comparing conditions with matched visual input but different behavioural response categories and vice versa. Overall, our results provide causal evidence for the involvement of the right IPS in different stages of sensory processing. Moreover, they also reflect the ability of the concurrent TMS-fMRI approach to divide a global task-related network into those elements that are specifically associated to the targeted area

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Effects of Attention on Multisensory Integration

The world presents information via a variety of sensory channels. To make sense of this information, we must determine what is relevant and ignore unhelpful noise. We then integrate congruent information within and across modalities to build coherent perceptions. Importantly, immediate goals and prevailing environmental factors may interact to affect our perceptual decisions. This dynamic process of multisensory integration is essential to successful perception in the real world, but can also lead to errors. The current project exploits some of these perceptual errors to explore how endogenous (task-directed) and exogenous (stimulus intensity) factors may influence multisensory integration. In a series of four experiments, we use the sound-induced flash illusion (SFI; Shams et al., 2000; 2002) and related audiovisual effects as indices of multisensory integration. Endogenous attention was manipulated using a focused attention visual task and a novel bimodal conditional attention task. In our first two experiments, we found that participants reported more illusions when attending to both sensory modalities. This effect was larger when the auditory stimuli were presented at near-threshold levels. Perceptual sensitivity (d′) was also found to decrease in the bimodal condition. We then manipulated auditory intensity in each of these tasks independently. Reports of the SFI were found to increase with the higher intensity auditory stimuli. However, differences in reporting these illusions within the same task were attributable to both changes in bias (c) and d′. Event-related potentials recorded in our first experiment revealed that the SFI was associated with smaller P3 potentials than found in valid targets. We also noted differences in the response-locked error positivity (Pe), with illusory stimuli having more positive amplitudes than real targets. However, the earlier occurring error-related negativity (ERN) was indistinguishable in real and illusory targets. This suggests that participants were less confident of the illusion during stimulus evaluation and one stage of response monitoring. We evaluate these results in terms of the directed attention and information reliability hypotheses (Andersen et al., 2004, 2005) and discuss how these and similar experiments may deepen our understanding of how multisensory perception is impacted at multiple stages of stimulus and response evaluation

Multisensory integration in complex rhythmic motor tasks

This publication-based thesis aimed to shed further light on the contributions of visual and auditory information to the execution of complex rhythmic motor tasks. To this end, a series of three empirical studies was designed. The first two studies were conducted in the experimental setting of long jumping – a task that does not only require high precision but also high velocity to leap as far as possible. Within the first study, we aimed to examine visual regulation during the long jump run-up using portable eye-tracking technology. Previous research on the topic of visual regulation in long jumping has identified a universal instance in participants’ gait behavior which introduces a rapid decrease in step variability. Since then, this parameter was considered synonymous with the (gait-based) onset of visual regulation. However, an investigation of actual visual behavior by means of direct measures was still pending to date. Hence, the first study was designed to identify a potential equivalent to the gait-based parameter within participants’ gaze behavior. Results revealed that the moment of the longest gaze on the take-off board coincided with the occurrence of the well-established gait parameter. Given that i) the gait-based visual regulation parameter has been associated with a reduction in footfall variability and ii) current research advocates the idea that fixations of longer duration facilitate motor actions, one might speculate that the longest gaze on the take-off board might as well serve the reduction of movement variability. While the parameter of visual regulation could be characterized in more detail within the first empirical investigation, the impact of other modalities has been neglected so far. However, current research is promoting the idea that, apart from visual information, auditory information might as well be crucial for the execution of motor tasks. ..

Functional imaging studies of visual-auditory integration in man.

This thesis investigates the central nervous system's ability to integrate visual and auditory information from the sensory environment into unified conscious perception. It develops the possibility that the principle of functional specialisation may be applicable in the multisensory domain. The first aim was to establish the neuroanatomical location at which visual and auditory stimuli are integrated in sensory perception. The second was to investigate the neural correlates of visual-auditory synchronicity, which would be expected to play a vital role in establishing which visual and auditory stimuli should be perceptually integrated. Four functional Magnetic Resonance Imaging studies identified brain areas specialised for: the integration of dynamic visual and auditory cues derived from the same everyday environmental events (Experiment 1), discriminating relative synchronicity between dynamic, cyclic, abstract visual and auditory stimuli (Experiment 2 & 3) and the aesthetic evaluation of visually and acoustically perceived art (Experiment 4). Experiment 1 provided evidence to suggest that the posterior temporo-parietal junction may be an important site of crossmodal integration. Experiment 2 revealed for the first time significant activation of the right anterior frontal operculum (aFO) when visual and auditory stimuli cycled asynchronously. Experiment 3 confirmed and developed this observation as the right aFO was activated only during crossmodal (visual-auditory), but not intramodal (visual-visual, auditory-auditory) asynchrony. Experiment 3 also demonstrated activation of the amygdala bilaterally during crossmodal synchrony. Experiment 4 revealed the neural correlates of supramodal, contemplative, aesthetic evaluation within the medial fronto-polar cortex. Activity at this locus varied parametrically according to the degree of subjective aesthetic beauty, for both visual art and musical extracts. The most robust finding of this thesis is that activity in the right aFO increases when concurrently perceived visual and auditory sensory stimuli deviate from crossmodal synchrony, which may veto the crossmodal integration of unrelated stimuli into unified conscious perception

The psychological and neurobiological determinants of social behavior:Assessing the affective, cognitive, and neural processes underlying trust and mentalizing

Much remains unknown about the specific affective, cognitive, and neural mechanisms involved that support our social behavior. The gap is addressed in my doctoral research in two ways: 1) by investigating the relationship between integral affect and social economic decisions and 2) by delineating how incidental affect impacts the neural signature of mentalizing. We first conduct a literature review and meta-analysis to inform the current debate concerning the determinants of trust. Our critical review shows that many experimental paradigms fall short of identifying clear affective mechanisms involved in trust. Therefore, we developed a novel experimental approach that relate the affective and cognitive reactions to betrayal to participants’ decision in the trust game and the dictator game. The results show that betrayal is positively and specifically associated with trust. Moreover, the perception of how socially distant we are from our interaction partners is significantly associated with trust. To assess the influence of incidental affect on the mentalizing network, we conducted an fMRI experiment using a novel false-belief task. The results revealed that activity in bilateral TPJ and IFG reflect false belief processing, and, at the same time as confirmed by conjunction analysis, suppressed by anxiety. During belief inferences threat specifically suppressed belief-based connectivity between putamen and its targets in IPS and dlPFC, and dispositional distress significantly modulated threat-related suppression of connectivity between the left TPJ seed and left IPS. These findings highlight the effects of both incidental and dispositional anxiety on specific nodes of the social cognition network