Neurocomputational mechanisms underlying cross-modal associations and their influence on perceptual decisions

When exposed to complementary features of information across sensory modalities, our brains formulate cross-modal associations between features of stimuli presented separately to multiple modalities. For example, auditory pitch-visual size associations map high-pitch tones with small-size visual objects, and low-pitch tones with large-size visual objects. Preferential, or congruent, cross-modal associations have been shown to affect behavioural performance, i.e. choice accuracy and reaction time (RT) across multisensory decision-making paradigms. However, the neural mechanisms underpinning such influences in perceptual decision formation remain unclear. Here, we sought to identify when perceptual improvements from associative congruency emerge in the brain during decision formation. In particular, we asked whether such improvements represent ‘early’ sensory processing benefits, or ‘late’ post-sensory changes in decision dynamics. Using a modified version of the Implicit Association Test (IAT), coupled with electroencephalography (EEG), we measured the neural activity underlying the effect of auditory stimulus-driven pitch-size associations on perceptual decision formation. Behavioural results showed that participants responded significantly faster during trials when auditory pitch was congruent, rather than incongruent, with its associative visual size counterpart. We used multivariate Linear Discriminant Analysis (LDA) to characterise the spatiotemporal dynamics of EEG activity underpinning IAT performance. We found an ‘Early’ component (∼100–110 ms post-stimulus onset) coinciding with the time of maximal discrimination of the auditory stimuli), and a ‘Late’ component (∼330–340 ms post-stimulus onset) underlying IAT performance. To characterise the functional role of these components in decision formation, we incorporated a neurally-informed Hierarchical Drift Diffusion Model (HDDM), revealing that the Late component decreases response caution, requiring less sensory evidence to be accumulated, whereas the Early component increased the duration of sensory-encoding processes for incongruent trials. Overall, our results provide a mechanistic insight into the contribution of ‘early’ sensory processing, as well as ‘late’ post-sensory neural representations of associative congruency to perceptual decision formation

Information-Theoretic Characterization of the Neural Mechanisms of Active Multisensory Decision Making

The signals delivered by different sensory modalities provide us with complementary information about the environment. A key component of interacting with the world is how to direct ones’ sensors so as to extract task-relevant information in order to optimize subsequent perceptual decisions. This process is often referred to as active sensing. Importantly, the processing of multisensory information acquired actively from multiple sensory modalities requires the interaction of multiple brain areas over time. Here we investigated the neural underpinnings of active visual-haptic integration during performance of a two-alternative forced choice (2AFC) reaction time (RT) task. We asked human subjects to discriminate the amplitude of two texture stimuli (a) using only visual (V) information, (b) using only haptic (H) information and (c) combining the two sensory cues (VH), while electroencephalograms (EEG) were recorded. To quantify multivariate interactions between EEG signals and active sensory experience in the three sensory conditions, we employed a novel information-theoretic methodology. This approach provides a principled way to quantify the contribution of each one of the sensory modalities to the perception of the stimulus and assess whether the respective neural representations may interact to form a percept of the stimulus and ultimately drive perceptual decisions. Application of this method to our data identified (a) an EEG component (comprising frontal and occipital electrodes) carrying behavioral information that is common to the two sensory inputs and (b) another EEG component (mainly motor) reflecting a synergistic representational interaction between the two sensory inputs. We suggest that the proposed approach can be used to elucidate the neural mechanisms underlying cross-modal interactions in active multisensory processing and decision-making

Applications of Information Theory to Analysis of Neural Data

Information theory is a practical and theoretical framework developed for the study of communication over noisy channels. Its probabilistic basis and capacity to relate statistical structure to function make it ideally suited for studying information flow in the nervous system. It has a number of useful properties: it is a general measure sensitive to any relationship, not only linear effects; it has meaningful units which in many cases allow direct comparison between different experiments; and it can be used to study how much information can be gained by observing neural responses in single trials, rather than in averages over multiple trials. A variety of information theoretic quantities are commonly used in neuroscience - (see entry "Definitions of Information-Theoretic Quantities"). In this entry we review some applications of information theory in neuroscience to study encoding of information in both single neurons and neuronal populations.Comment: 8 pages, 2 figure

Erratum to: 36th International Symposium on Intensive Care and Emergency Medicine

[This corrects the article DOI: 10.1186/s13054-016-1208-6.]