The multisensory function of the human primary visual cortex

It has been nearly 10 years since Ghazanfar and Schroeder (2006) proposed that the neocortex is essentially multisensory in nature. However, it is only recently that sufficient and hard evidence that supports this proposal has accrued. We review evidence that activity within the human primary visual cortex plays an active role in multisensory processes and directly impacts behavioural outcome. This evidence emerges from a full pallet of human brain imaging and brain mapping methods with which multisensory processes are quantitatively assessed by taking advantage of particular strengths of each technique as well as advances in signal analyses. Several general conclusions about multisensory processes in primary visual cortex of humans are supported relatively solidly. First, haemodynamic methods (fMRI/PET) show that there is both convergence and integration occurring within primary visual cortex. Second, primary visual cortex is involved in multisensory processes during early post-stimulus stages (as revealed by EEG/ERP/ERFs as well as TMS). Third, multisensory effects in primary visual cortex directly impact behaviour and perception, as revealed by correlational (EEG/ERPs/ERFs) as well as more causal measures (TMS/tACS). While the provocative claim of Ghazanfar and Schroeder (2006) that the whole of neocortex is multisensory in function has yet to be demonstrated, this can now be considered established in the case of the human primary visual cortex

The development of attentional control mechanisms in multisensory environments

Outside the laboratory, people need to pay attention to relevant objects that are typically multisensory, but it remains poorly understood how the underlying neurocognitive mechanisms develop. We investigated when adult-like mechanisms controlling one’s attentional selection of visual and multisensory objects emerge across childhood. Five-, 7-, and 9-year-olds were compared with adults in their performance on a computer game-like multisensory spatial cueing task, while 129-channel EEG was simultaneously recorded. Markers of attentional control were behavioural spatial cueing effects and the N2pc ERP component (analysed traditionally and using a multivariate electrical neuroimaging framework). In behaviour, adult-like visual attentional control was present from age 7 onwards, whereas multisensory control was absent in all children groups. In EEG, multivariate analyses of the activity over the N2pc time-window revealed stable brain activity patterns in children. Adult-like visual-attentional control EEG patterns were present age 7 onwards, while multisensory control activity patterns were found in 9-year-olds (albeit behavioural measures showed no effects). By combining rigorous yet naturalistic paradigms with multivariate signal analyses, we demonstrated that visual attentional control seems to reach an adult-like state at ∼7 years, before adult-like multisensory control, emerging at ∼9 years. These results enrich our understanding of how attention in naturalistic settings develops

Mental Rotation of Digitally-Rendered Haptic Objects

Sensory substitution is an effective means to rehabilitate many visual functions after visual impairment or blindness. Tactile information, for example, is particularly useful for functions such as reading, mental rotation, shape recognition, or exploration of space. Extant haptic technologies typically rely on real physical objects or pneumatically driven renderings and thus provide a limited library of stimuli to users. New developments in digital haptic technologies now make it possible to actively simulate an unprecedented range of tactile sensations. We provide a proof-of-concept for a new type of technology (hereafter haptic tablet) that renders haptic feedback by modulating the friction of a flat screen through ultrasonic vibrations of varying shapes to create the sensation of texture when the screen is actively explored. We reasoned that participants should be able to create mental representations of letters presented in normal and mirror-reversed haptic form without the use of any visual information and to manipulate such representations in a mental rotation task. Healthy sighted, blindfolded volunteers were trained to discriminate between two letters (either L and P, or F and G; counterbalanced across participants) on a haptic tablet. They then tactually explored all four letters in normal or mirror-reversed form at different rotations (0°, 90°, 180°, and 270°) and indicated letter form (i.e., normal or mirror-reversed) by pressing one of two mouse buttons. We observed the typical effect of rotation angle on object discrimination performance (i.e., greater deviation from 0° resulted in worse performance) for trained letters, consistent with mental rotation of these haptically-rendered objects. We likewise observed generally slower and less accurate performance with mirror-reversed compared to prototypically oriented stimuli. Our findings extend existing research in multisensory object recognition by indicating that a new technology simulating active haptic feedback can support the generation and spatial manipulation of mental representations of objects. Thus, such haptic tablets can offer a new avenue to mitigate visual impairments and train skills dependent on mental object-based representations and their spatial manipulation

The COGs (context, object, and goals) in multisensory processing

Our understanding of how perception operates in real-world environments has been substantially advanced by studying both multisensory processes and “top-down” control processes influencing sensory processing via activity from higher-order brain areas, such as attention, memory, and expectations. As the two topics have been traditionally studied separately, the mechanisms orchestrating real-world multisensory processing remain unclear. Past work has revealed that the observer’s goals gate the influence of many multisensory processes on brain and behavioural responses, whereas some other multisensory processes might occur independently of these goals. Consequently, other forms of top-down control beyond goal dependence are necessary to explain the full range of multisensory effects currently reported at the brain and the cognitive level. These forms of control include sensitivity to stimulus context as well as the detection of matches (or lack thereof) between a multisensory stimulus and categorical attributes of naturalistic objects (e.g. tools, animals). In this review we discuss and integrate the existing findings that demonstrate the importance of such goal-, object- and context-based top-down control over multisensory processing. We then put forward a few principles emerging from this literature review with respect to the mechanisms underlying multisensory processing and discuss their possible broader implications

Gelesen & Kommentiert «Binokulare Amblyopiebehandlung verbessert die manuelle Geschicklichkeit» [ Read & Reviewed “Binocular amblyopia treatment improves manual dexterity” ]

A review and a commentary on the article of Birch et al.'s (2023) “Binocular amblyopia treatment improves manual dexterity" for the journal PhysioScience. The original langauge of the current manuscript is German; an English translation has been included in this preprint

The threat of paper mills to social science journals: The case of the Tanu.pro paper mill in Mind, Brain & Education

Fraudulent published papers used to be thought to be rare, but in recent years there has been growing awareness of coordinated activities by organizations that charge authors a fee to plant articles in reputable journals. These are known as paper mills. We reflect here on how Mind, Brain and Education suffered from such an attack in 2022 and 2023, discussing what we have learnt from this experience about how paper mills might operate in social sciences, how to spot signs that an article is fraudulent, and what steps might be taken to prevent such attacks in future

Expert Attention: Attentional allocation depends on the differential development of multisensory number representations

Traditional models developed within cognitive psychology suggest that attention is deployed flexibly and irrespective of differences in expertise with to-be-attended stimuli. However, everyday environments are inherently multisensory and observers differ in familiarity with particular unisensory representations (e.g., number words, in contrast with digits). To test whether the predictions of the traditional models extend to such naturalistic settings, six-year-olds, 11-year-olds and young adults (N=83) searched for predefined numerals amongst a small or large number of distractor digits, while distractor number words, digits or their combination were presented peripherally. Concurrently presented number words and audiovisual stimuli that were compatible with the target digit facilitated young children’s selective attention. In contrast, for older children and young adults number words and audiovisual stimuli that were incompatible with their visual targets resulted in a cost on reaction time. These findings suggest that multisensory and familiarity-based influences interact dynamically as they shape selective attention. Therefore, models of selective attention should include multisensory and familiarity-dependent constraints

Uncovering the mechanisms of real-world attentional control over the course of primary education

Schooling may shape children’s abilities to control their attention, but it is unclear if this impact extends from control over visual objects to encompass multisensory objects, which are more typical of everyday environments. We compared children across three primary school grades (Swiss 1st, 3rd, and 5th grade) on their performance on a computer game-like audio-visual attentional control task, while recording their EEG. Behavioural markers of visual attentional control were present from 3rd grade (after 2 years of schooling), whereas multisensory attentional control was not detected in any group. However, multivariate whole-brain EEG analyses (‘electrical neuroimaging’) revealed stable patterns of brain activity that indexed both types of attentional control – visual control in all groups, and multisensory attentional control – from 3rd grade onwards. Our findings suggest that using multivariate EEG approaches can uncover otherwise undetectable mechanisms of attentional control over visual and multisensory objects and characterise how they differ at different educational stages