23,811 research outputs found
Contributions of local speech encoding and functional connectivity to audio-visual speech perception
Seeing a speaker’s face enhances speech intelligibility in adverse environments. We investigated the underlying network mechanisms by quantifying local speech representations and directed connectivity in MEG data obtained while human participants listened to speech of varying acoustic SNR and visual context. During high acoustic SNR speech encoding by temporally entrained brain activity was strong in temporal and inferior frontal cortex, while during low SNR strong entrainment emerged in premotor and superior frontal cortex. These changes in local encoding were accompanied by changes in directed connectivity along the ventral stream and the auditory-premotor axis. Importantly, the behavioral benefit arising from seeing the speaker’s face was not predicted by changes in local encoding but rather by enhanced functional connectivity between temporal and inferior frontal cortex. Our results demonstrate a role of auditory-frontal interactions in visual speech representations and suggest that functional connectivity along the ventral pathway facilitates speech comprehension in multisensory environments
The nature of the animacy organization in human ventral temporal cortex
The principles underlying the animacy organization of the ventral temporal
cortex (VTC) remain hotly debated, with recent evidence pointing to an animacy
continuum rather than a dichotomy. What drives this continuum? According to the
visual categorization hypothesis, the continuum reflects the degree to which
animals contain animal-diagnostic features. By contrast, the agency hypothesis
posits that the continuum reflects the degree to which animals are perceived as
(social) agents. Here, we tested both hypotheses with a stimulus set in which
visual categorizability and agency were dissociated based on representations in
convolutional neural networks and behavioral experiments. Using fMRI, we found
that visual categorizability and agency explained independent components of the
animacy continuum in VTC. Modeled together, they fully explained the animacy
continuum. Finally, clusters explained by visual categorizability were
localized posterior to clusters explained by agency. These results show that
multiple organizing principles, including agency, underlie the animacy
continuum in VTC.Comment: 16 pages, 5 figures, code+data at -
https://doi.org/10.17605/OSF.IO/VXWG9 Update - added supplementary results
and edited abstrac
Unmasking Clever Hans Predictors and Assessing What Machines Really Learn
Current learning machines have successfully solved hard application problems,
reaching high accuracy and displaying seemingly "intelligent" behavior. Here we
apply recent techniques for explaining decisions of state-of-the-art learning
machines and analyze various tasks from computer vision and arcade games. This
showcases a spectrum of problem-solving behaviors ranging from naive and
short-sighted, to well-informed and strategic. We observe that standard
performance evaluation metrics can be oblivious to distinguishing these diverse
problem solving behaviors. Furthermore, we propose our semi-automated Spectral
Relevance Analysis that provides a practically effective way of characterizing
and validating the behavior of nonlinear learning machines. This helps to
assess whether a learned model indeed delivers reliably for the problem that it
was conceived for. Furthermore, our work intends to add a voice of caution to
the ongoing excitement about machine intelligence and pledges to evaluate and
judge some of these recent successes in a more nuanced manner.Comment: Accepted for publication in Nature Communication
Quantifying the Performance of Explainability Algorithms
Given the complexity of the deep neural network (DNN), DNN has long been criticized for its lack of interpretability in its decision-making process. This 'black box' nature has been preventing the adaption of DNN in life-critical tasks. In recent years, there has been a surge of interest around the concept of artificial intelligence explainability/interpretability (XAI), where the goal is to produce an interpretation for a decision made by a DNN algorithm. While many explainability algorithms have been proposed for peaking into the decision-making process of DNN, there has been a limited exploration into the assessment of the performance of explainability methods, with most evaluations centred around subjective human visual perception of the produced interpretations. In this study, we explore a more objective strategy for quantifying the performance of explainability algorithms on DNNs. More specifically, we propose two quantitative performance metrics: i) \textbf{Impact Score} and ii) \textbf{Impact Coverage}. Impact Score assesses the percentage of critical factors with either strong confidence reduction impact or decision shifting impact. Impact Coverage accesses the percentage overlapping of adversarially impacted factors in the input. Furthermore, a comprehensive analysis using this approach was conducted on several explainability methods (LIME, SHAP, and Expected Gradients) on different task domains, such as visual perception, speech recognition and natural language processing (NLP). The empirical evidence suggests that there is significant room for improvement for all evaluated explainability methods. At the same time, the evidence also suggests that even the latest explainability methods can not produce steady better results across different task domains and different test scenarios
Beyond saliency: understanding convolutional neural networks from saliency prediction on layer-wise relevance propagation
Despite the tremendous achievements of deep convolutional neural networks
(CNNs) in many computer vision tasks, understanding how they actually work
remains a significant challenge. In this paper, we propose a novel two-step
understanding method, namely Salient Relevance (SR) map, which aims to shed
light on how deep CNNs recognize images and learn features from areas, referred
to as attention areas, therein. Our proposed method starts out with a
layer-wise relevance propagation (LRP) step which estimates a pixel-wise
relevance map over the input image. Following, we construct a context-aware
saliency map, SR map, from the LRP-generated map which predicts areas close to
the foci of attention instead of isolated pixels that LRP reveals. In human
visual system, information of regions is more important than of pixels in
recognition. Consequently, our proposed approach closely simulates human
recognition. Experimental results using the ILSVRC2012 validation dataset in
conjunction with two well-established deep CNN models, AlexNet and VGG-16,
clearly demonstrate that our proposed approach concisely identifies not only
key pixels but also attention areas that contribute to the underlying neural
network's comprehension of the given images. As such, our proposed SR map
constitutes a convenient visual interface which unveils the visual attention of
the network and reveals which type of objects the model has learned to
recognize after training. The source code is available at
https://github.com/Hey1Li/Salient-Relevance-Propagation.Comment: 35 pages, 15 figure
Contribution of Color Information in Visual Saliency Model for Videos
International audienceMuch research has been concerned with the contribution of the low level features of a visual scene to the deployment of visual attention. Bottom-up saliency models have been developed to predict the location of gaze according to these features. So far, color besides to brightness, contrast and motion is considered as one of the primary features in computing bottom-up saliency. However, its contribution in guiding eye movements when viewing natural scenes has been debated. We investigated the contribution of color information in a bottom-up visual saliency model. The model efficiency was tested using the experimental data obtained on 45 observers who were eye tracked while freely exploring a large data set of color and grayscale videos. The two datasets of recorded eye positions, for grayscale and color videos, were compared with a luminance-based saliency model. We incorporated chrominance information to the model. Results show that color information improves the performance of the saliency model in predicting eye positions
- …