Making sense of real-world scenes

To interact with the world, we have to make sense of the continuous sensory input conveying information about our environment. A recent surge of studies has investigated the processes enabling scene understanding, using increasingly complex stimuli and sophisticated analyses to highlight the visual features and brain regions involved. However, there are two major challenges to producing a comprehensive framework for scene understanding. First, scene perception is highly dynamic, subserving multiple behavioral goals. Second, a multitude of different visual properties co-occur across scenes and may be correlated or independent. We synthesize the recent literature and argue that for a complete view of scene understanding, it is necessary to account for both differing observer goals and the contribution of diverse scene properties

Neural representation of geometry and surface properties in object and scene perception

Multiple cortical regions are crucial for perceiving the visual world, yet the processes shaping representations in these regions are unclear. To address this issue, we must elucidate how perceptual features shape representations of the environment. Here, we explore how the weighting of different visual features affects neural representations of objects and scenes, focusing on the scene-selective parahippocampal place area (PPA), but additionally including the retrosplenial complex (RSC), occipital place area (OPA), lateral occipital (LO) area, fusiform face area (FFA) and occipital face area (OFA). Across three experiments, we examined functional magnetic resonance imaging (fMRI) activity while human observers viewed scenes and objects that varied in geometry (shape/layout) and surface properties (texture/material). Interestingly, we found equal sensitivity in the PPA for these properties within a scene, revealing that spatial-selectivity alone does not drive activation within this cortical region. We also observed sensitivity to object texture in PPA, but not to the same degree as scene texture, and representations in PPA varied when objects were placed within scenes. We conclude that PPA may process surface properties in a domain-specific manner, and that the processing of scene texture and geometry is equally-weighted in PPA and may be mediated by similar underlying neuronal mechanisms

Diagnostic information use to understand brain mechanisms of facial expression categorization

Proficient categorization of facial expressions is crucial for normal social interaction. Neurophysiological, behavioural, event-related potential, lesion and functional neuroimaging techniques can be used to investigate the underlying brain mechanisms supporting this seemingly effortless process, and the associated arrangement of bilateral networks. These brain areas exhibit consistent and replicable activation patterns, and can be broadly defined to include visual (occipital and temporal), limbic (amygdala) and prefrontal (orbitofrontal) regions. Together, these areas support early perceptual processing, the formation of detailed representations and subsequent recognition of expressive faces. Despite the critical role of facial expressions in social communication and extensive work in this area, it is still not known how the brain decodes nonverbal signals in terms of expression-specific features. For these reasons, this thesis investigates the role of these so-called diagnostic facial features at three significant stages in expression recognition; the spatiotemporal inputs to the visual system, the dynamic integration of features in higher visual (occipitotemporal) areas, and early sensitivity to features in V1. In Chapter 1, the basic emotion categories are presented, along with the brain regions that are activated by these expressions. In line with this, the current cognitive theory of face processing reviews functional and anatomical dissociations within the distributed neural “face network”. Chapter 1 also introduces the way in which we measure and use diagnostic information to derive brain sensitivity to specific facial features, and how this is a useful tool by which to understand spatial and temporal organisation of expression recognition in the brain. In relation to this, hierarchical, bottom-up neural processing is discussed along with high-level, top-down facilitatory mechanisms. Chapter 2 describes an eye-movement study that reveals inputs to the visual system via fixations reflect diagnostic information use. Inputs to the visual system dictate the information distributed to cognitive systems during the seamless and rapid categorization of expressive faces. How we perform eye-movements during this task informs how task-driven and stimulus-driven mechanisms interact to guide the extraction of information supporting recognition. We recorded eye movements of observers who categorized the six basic categories of facial expressions. We use a measure of task-relevant information (diagnosticity) to discuss oculomotor behaviour, with focus on two findings. Firstly, fixated regions reveal expression differences. Secondly, by examining fixation sequences, the intersection of fixations with diagnostic information increases in a sequence of fixations. This suggests a top-down drive to acquire task-relevant information, with different functional roles for first and final fixations. A combination of psychophysical studies of visual recognition together with the EEG (electroencephalogram) signal is used to infer the dynamics of feature extraction and use during the recognition of facial expressions in Chapter 3. The results reveal a process that integrates visual information over about 50 milliseconds prior to the face-sensitive N170 event-related potential, starting at the eye region, and proceeding gradually towards lower regions. The finding that informative features for recognition are not processed simultaneously but in an orderly progression over a short time period is instructive for understanding the processes involved in visual recognition, and in particular the integration of bottom-up and top-down processes. In Chapter 4 we use fMRI to investigate the task-dependent activation to diagnostic features in early visual areas, suggesting top-down mechanisms as V1 traditionally exhibits only simple response properties. Chapter 3 revealed that diagnostic features modulate the temporal dynamics of brain signals in higher visual areas. Within the hierarchical visual system however, it is not known if an early (V1/V2/V3) sensitivity to diagnostic information contributes to categorical facial judgements, conceivably driven by top-down signals triggered in visual processing. Using retinotopic mapping, we reveal task-dependent information extraction within the earliest cortical representation (V1) of two features known to be differentially necessary for face recognition tasks (eyes and mouth). This strategic encoding of face images is beyond typical V1 properties and suggests a top-down influence of task extending down to the earliest retinotopic stages of visual processing. The significance of these data is discussed in the context of the cortical face network and bidirectional processing in the visual system. The visual cognition of facial expression processing is concerned with the interactive processing of bottom-up sensory-driven information and top-down mechanisms to relate visual input to categorical judgements. The three experiments presented in this thesis are summarized in Chapter 5 in relation to how diagnostic features can be used to explore such processing in the human brain leading to proficient facial expression categorization

The Role of Diagnostic Objects in the Temporal Dynamics of Visual Scene Categorization

When we look at the world around us we are able to effortlessly categorize scenes, but it is still unclear what mechanisms we use to do so. Categorization could be driven by objects, low-level features, or a mixture of both. This study investigated the ways in which diagnostic objects (those found nearly exclusively in one scene category) contribute to scene categorization. It paired Electroencephalography (EEG) with machine learning classification to provide detailed temporal information about when categorization occurs. While recording EEG, participants categorized real-world photographs as one of three indoor scene types (bathroom, kitchen, office). They were shown either original images or versions where diagnostic or random objects had been obscured via localized Fourier phase randomization. EEG voltages and the independent components (ICs) of a whole brain independent component analysis (ICA) were used as feature vectors for a linear support vector machine (SVM) classifier to determine time-resolved accuracy. There were no significant differences in decoding accuracy between categories or between diagnostic and random conditions. Poor classifier performance is likely due to a lack of power, or overfitting of the model. It could also reflect unclear EEG-based neural correlates of each scene type due to the inherent similarities in the categories. While the lack of significant decoding makes it difficult to make strong conclusions about the role of diagnostic objects in visual scene categorization, this study addresses important considerations for pairing EEG with decoding techniques and highlights some of the broader difficulties of isolating distinct features of visual scenes

Diagnostic colours of emotions

This thesis investigates the role of colour in the cognitive processesing of emotional information. The research is guided by the effect of colour diagnosticity which has been shown previously to influence recognition performance of several types of objects as well as natural scenes. The research presented in Experiment 1 examined whether colour information is considered a diagnostic perceptual feature of seven emotional categories: happiness, sadness, anger, fear, disgust, surprise and neutral. Participants (N = 119), who were naïve to the specific purpose and expectations of the experiment, chose colour more than any other perceptual quality (e.g. shape and tactile information) as a feature that describes the seven emotional categories. The specific colour features given for the six basic emotions were consistently different from those given to the non-emotional neutral category. While emotional categories were often described by chromatic colour features (e.g. red, blue, orange) the neutral category was often ascribed achromatic colour features (e.g. white, grey, transparent) as the most symptomatic perceptual qualities for its description. The emotion 'anger' was unique in being the only emotion showing an agreement higher that 50% of the total given colour features for one particular colour - red. Confirming that colour is a diagnostic feature of emotions led to the examination of the effect of diagnostic colours of emotion on recognition memory for emotional words and faces: the effect, if any, of appropriate and inappropriate colours (matched with emotion) on the strength of memory for later recognition of faces and words (Experiments 2 & 3). The two experiments used retention intervals of 15 minutes and one week respectively and the colour-emotion associations were determined for each individual participant. Results showed that regardless of the subject’s consistency level in associating colours with emotions, and compared with the individual inappropriate or random colours, individual appropriate colours of emotions significantly enhance recognition memory for six basic emotional faces and words. This difference between the individual inappropriate colours or random colours and the individual appropriate colours of emotions was not found to be significant for non-emotional neutral stimuli. Post hoc findings from both experiments further show that appropriate colours of emotion are associated more consistently than inappropriate colours of emotions. This suggests that appropriate colour-emotion associations are unique both in their strength of association and in the form of their representation. Experiment 4 therefore aimed to investigate whether appropriate colour-emotion associations also trigger an implicit automatic cognitive system that allows faster naming times for appropriate versus inappropriate colours of emotional word carriers. Results from the combined Emotional-Semantic Stroop task confirm the above hypothesis and therefore imply that colour plays a substantial role not only in our conceptual representations of objects but also in our conceptual representations of basic emotions. The resemblance of the present findings collectively to those found previously for objects and natural scenes suggests a common cognitive mechanism for the processing of emotional diagnostic colours and the processing of diagnostic colours of objects or natural scenes. Overall, this thesis provides the foundation for many future directions of research in the area of colour and emotion as well as a few possible immediate practical implications

Conceptual Flexibility: Behavioral and Neural Variations in Semantic Memory Retrieval

Understanding the neural organization of semantic memory - our shared general knowledge - has the potential to uncover the neural mechanisms by which we give meaning to the endless array of objects that we encounter in the world. One prominent set of theories posits a distributed organization of semantic memory - remembering object features activates brain regions overlapping with or adjacent to regions involved in perceiving and acting on those features (e.g., Allport, 1985). Despite accumulated evidence favoring these theories, it is important to understand both the commonalities in the mapping between perception and memory, as well as meaningful variability across sources such as contexts, people, and use. In three studies, we found evidence that while conceptual knowledge is grounded in neural substrates, several factors contribute to variations in semantic memory retrieval. In Chapter 2, we used the logic often used in neuroimaging studies of semantic memory by demonstrating overlapping chromaticity effects (e.g., greater response to colored than grayscale stimuli) in the left lingual gyrus for both color perception and color knowledge. Chapter 3 investigated whether the mapping between perception and memory varied across contexts and participants. Whereas context (here, fidelity of color information as manipulated through task demand) varied the extent to which the left fusiform gyrus was active during a color similarity judgment, individual differences in cognitive style predicted activity in the left lingual gyrus. We replicated these results in a second experiment that controlled for stimulus modality and anticipatory strategies. In Chapter 4, we used a training paradigm to investigate the role of feature diagnosticity (i.e., features that best distinguish between two otherwise similar categories) in semantic representations. Whereas subjects had knowledge of feature importance in novel object categorization, whether they used this information affected neural representations. Ventral temporal brain regions were more active during a separate retrieval task for subjects who learned and used the diagnostic feature for object categorization. Additionally, behavioral ratings of similarity predicted multivariate neural similarity. Collectively, this work suggests that semantic representations, integral to a memory system often thought of as free of contextual constraints, contain meaningful variations across contexts, people, and use

Colour and Naming in Healthy and Aphasic People

Abstract The purpose of this study was to create a paradigm suitable for people with aphasia and healthy subjects to evaluate the influence of colour on naming pictures of objects. We designed a completely new stimulus set based on images of 140 common real objects that were inspired by the Snodgrass and Vanderwart picture set (1980). We were especially interested whether there is a difference in performance between the aphasic patients and the group of healthy controls. Adding chromatic information to pictures of objects shows only a small effect in verification and categorisation tasks. However, when observers are required to name objects, colour speeds performance and enhances accuracy (Rossion & Pourtois, 2004). The present study contrasts two different claims as to why colour may benefit object naming. The first is that colour simply aids the segmentation of the object from its background (Wichmann et al., 2002). The second is that colour may help to elicit a wider range of associations with the object, thereby enhancing lexical access (Bisiach, 1966). To distinguish between these processes an equal number of pictures containing high and low colour diagnostic objects were presented against either fractal noise or uniform backgrounds in a naming task to aphasic subjects with anomia and to healthy controls. Performance for chromatic stimuli was compared with that for monochrome stimuli equated in luminance. Results show that colour facilitates naming significantly in both subject groups and there was no significant difference between objects with high or low colour diagnostic values. We also found that object segmentation and the lexical access seem to occur in parallel processes, rather than in an additive way

Decoding face categories in diagnostic subregions of primary visual cortex

Higher visual areas in the occipitotemporal cortex contain discrete regions for face processing, but it remains unclear if V1 is modulated by top-down influences during face discrimination, and if this is widespread throughout V1 or localized to retinotopic regions processing task-relevant facial features. Employing functional magnetic resonance imaging (fMRI), we mapped the cortical representation of two feature locations that modulate higher visual areas during categorical judgements – the eyes and mouth. Subjects were presented with happy and fearful faces, and we measured the fMRI signal of V1 regions processing the eyes and mouth whilst subjects engaged in gender and expression categorization tasks. In a univariate analysis, we used a region-of-interest-based general linear model approach to reveal changes in activation within these regions as a function of task. We then trained a linear pattern classifier to classify facial expression or gender on the basis of V1 data from ‘eye’ and ‘mouth’ regions, and from the remaining non-diagnostic V1 region. Using multivariate techniques, we show that V1 activity discriminates face categories both in local ‘diagnostic’ and widespread ‘non-diagnostic’ cortical subregions. This indicates that V1 might receive the processed outcome of complex facial feature analysis from other cortical (i.e. fusiform face area, occipital face area) or subcortical areas (amygdala)

Interpreting EEG and MEG signal modulation in response to facial features: the influence of top-down task demands on visual processing strategies

The visual processing of faces is a fast and efficient feat that our visual system usually accomplishes many times a day. The N170 (an Event-Related Potential) and the M170 (an Event-Related Magnetic Field) are thought to be prominent markers of the face perception process in the ventral stream of visual processing that occur ~ 170 ms after stimulus onset. The question of whether face processing at the time window of the N170 and M170 is automatically driven by bottom-up visual processing only, or whether it is also modulated by top-down control, is still debated in the literature. However, it is known from research on general visual processing, that top-down control can be exerted much earlier along the visual processing stream than the N170 and M170 take place. I conducted two studies, each consisting of two face categorization tasks. In order to examine the influence of top-down control on the processing of faces, I changed the task demands from one task to the next, while presenting the same set of face stimuli. In the first study, I recorded participants’ EEG signal in response to faces while they performed both a Gender task and an Expression task on a set of expressive face stimuli. Analyses using Bubbles (Gosselin & Schyns, 2001) and Classification Image techniques revealed significant task modulations of the N170 ERPs (peaks and amplitudes) and the peak latency of maximum information sensitivity to key facial features. However, task demands did not change the information processing during the N170 with respect to behaviourally diagnostic information. Rather, the N170 seemed to integrate gender and expression diagnostic information equally in both tasks. In the second study, participants completed the same behavioural tasks as in the first study (Gender and Expression), but this time their MEG signal was recorded in order to allow for precise source localisation. After determining the active sources during the M170 time window, a Mutual Information analysis in connection with Bubbles was used to examine voxel sensitivity to both the task-relevant and the task-irrelevant face category. When a face category was relevant for the task, sensitivity to it was usually higher and peaked in different voxels than sensitivity to the task-irrelevant face category. In addition, voxels predictive of categorization accuracy were shown to be sensitive to task-relevant, behaviourally diagnostic facial features only. I conclude that facial feature integration during both N170 and M170 is subject to top-down control. The results are discussed against the background of known face processing models and current research findings on visual processing