Multimodal Computational Attention for Scene Understanding

Robotic systems have limited computational capacities. Hence, computational attention models are important to focus on specific stimuli and allow for complex cognitive processing. For this purpose, we developed auditory and visual attention models that enable robotic platforms to efficiently explore and analyze natural scenes. To allow for attention guidance in human-robot interaction, we use machine learning to integrate the influence of verbal and non-verbal social signals into our models

A Psychophysical Oriented Saliency Map Prediction Model

Visual attention is one of the most significant characteristics for selecting and understanding the outside redundancy world. The human vision system cannot process all information simultaneously, due to the visual information bottleneck. In order to reduce the redundant input of visual information, the human visual system mainly focuses on dominant parts of scenes. This is commonly known as visual saliency map prediction. This paper proposed a new psychophysical saliency prediction architecture, WECSF, inspired by multi-channel model of visual cortex functioning in humans. The model consists of opponent color channels, wavelet transform, wavelet energy map, and contrast sensitivity function for extracting low-level image features and providing maximum approximation to the human visual system. The proposed model is evaluated using several datasets, including the MIT1003, MIT300, TORONTO, SID4VAM, and UCF Sports datasets. We also quantitatively and qualitatively compare the saliency prediction performance with that of other state-of-the-art models. Our model achieved strongly stable and better performance with different metrics on nature images, psychophysical synthetic images and dynamic videos. Additionally, we found that Fourier and spectral-inspired saliency prediction models outperformed other state-of-the-art non-neural network and even deep neural network models on psychophysical synthetic images, it can be explained and supported the Fourier Vision Hypothesis. Finally, the proposed model could be used as a computational model of primate vision system and help us understand mechanism of vision system

Improving ecological forecasts using model and data constraints

Terrestrial ecosystems are essential to human well-being, but their future remains highly uncertain, as evidenced by the huge disparities in model projections of the land carbon sink. The existence of these disparities despite the recent explosion of novel data streams, including the TRY plant traits database, the Landsat archive, and global eddy covariance tower networks, suggests that these data streams are not being utilized to their full potential by the terrestrial ecosystem modeling community. Therefore, the overarching objective of my dissertation is to identify how these various data streams can be used to improve the precision of model predictions by constraining model parameters. In chapter 1, I use a hierarchical multivariate meta-analysis of the TRY database to assess the dependence of trait correlations on ecological scale and evaluate the utility of these correlations for constraining ecosystem model parameters. I find that global trait correlations are generally consistent within plant functional types, and leveraging the multivariate trait space is an effective way to constrain trait estimates for data-limited traits and plant functional types. My next two chapters assess the ability to measure traits using remote sensing by exploring the links between leaf traits and reflectance spectra. In chapter 2, I introduce a method for estimating traits from spectra via radiative transfer model inversion. I then use this approach to show that although the precise location, width, and quantity of spectral bands significantly affects trait retrieval accuracy, a wide range of sensor configurations are capable of providing trait information. In chapter 3, I apply this approach to a large database of leaf spectra to show that traits vary as much within as across species, and much more across species within a functional type than across functional types. Finally, in chapter 4, I synthesize the findings of the previous chapters to calibrate a vegetation model's representation of canopy radiative transfer against observed remotely-sensed surface reflectance. Although the calibration successfully constrained canopy structural parameters, I identify issues with model representations of wood and soil reflectance that inhibit its ability to accurately reproduce remote sensing observations

Anterior insula coordinates hierarchical processing of tactile mismatch responses

The body underlies our sense of self, emotion, and agency. Signals arising from the skin convey warmth, social touch, and the physical characteristics of external stimuli. Surprising or unexpected tactile sensations can herald events of motivational salience, including imminent threats (e.g., an insect bite) and hedonic rewards (e.g., a caressing touch). Awareness of such events is thought to depend upon the hierarchical integration of body-related mismatch responses by the anterior insula. To investigate this possibility, we measured brain activity using functional magnetic resonance imaging, while healthy participants performed a roving tactile oddball task. Mass-univariate analysis demonstrated robust activations in limbic, somatosensory, and prefrontal cortical areas previously implicated in tactile deviancy, body awareness, and cognitive control. Dynamic Causal Modelling revealed that unexpected stimuli increased the strength of forward connections along a caudal to rostral hierarchy-projecting from thalamic and somatosensory regions towards insula, cingulate and prefrontal cortices. Within this ascending flow of sensory information, the AIC was the only region to show increased backwards connectivity to the somatosensory cortex, augmenting a reciprocal exchange of neuronal signals. Further, participants who rated stimulus changes as easier to detect showed stronger modulation of descending PFC to AIC connections by deviance. These results suggest that the AIC coordinates hierarchical processing of tactile prediction error. They are interpreted in support of an embodied predictive coding model where AIC mediated body awareness is involved in anchoring a global neuronal workspace

Saliency in Augmented Reality

With the rapid development of multimedia technology, Augmented Reality (AR) has become a promising next-generation mobile platform. The primary theory underlying AR is human visual confusion, which allows users to perceive the real-world scenes and augmented contents (virtual-world scenes) simultaneously by superimposing them together. To achieve good Quality of Experience (QoE), it is important to understand the interaction between two scenarios, and harmoniously display AR contents. However, studies on how this superimposition will influence the human visual attention are lacking. Therefore, in this paper, we mainly analyze the interaction effect between background (BG) scenes and AR contents, and study the saliency prediction problem in AR. Specifically, we first construct a Saliency in AR Dataset (SARD), which contains 450 BG images, 450 AR images, as well as 1350 superimposed images generated by superimposing BG and AR images in pair with three mixing levels. A large-scale eye-tracking experiment among 60 subjects is conducted to collect eye movement data. To better predict the saliency in AR, we propose a vector quantized saliency prediction method and generalize it for AR saliency prediction. For comparison, three benchmark methods are proposed and evaluated together with our proposed method on our SARD. Experimental results demonstrate the superiority of our proposed method on both of the common saliency prediction problem and the AR saliency prediction problem over benchmark methods. Our data collection methodology, dataset, benchmark methods, and proposed saliency models will be publicly available to facilitate future research