Revealing the information contents of memory within the stimulus information representation framework

The information contents of memory are the cornerstone of the most influential models in cognition. To illustrate, consider that in predictive coding, a prediction implies that specific information is propagated down from memory through the visual hierarchy. Likewise, recognizing the input implies that sequentially accrued sensory evidence is successfully matched with memorized information (categorical knowledge). Although the existing models of prediction, memory, sensory representation and categorical decision are all implicitly cast within an information processing framework, it remains a challenge to precisely specify what this information is, and therefore where, when and how the architecture of the brain dynamically processes it to produce behaviour. Here, we review a framework that addresses these challenges for the studies of perception and categorization–stimulus information representation (SIR). We illustrate how SIR can reverse engineer the information contents of memory from behavioural and brain measures in the context of specific cognitive tasks that involve memory. We discuss two specific lessons from this approach that generally apply to memory studies: the importance of task, to constrain what the brain does, and of stimulus variations, to identify the specific information contents that are memorized, predicted, recalled and replayed

Making sense of real-world scenes

To interact with the world, we have to make sense of the continuous sensory input conveying information about our environment. A recent surge of studies has investigated the processes enabling scene understanding, using increasingly complex stimuli and sophisticated analyses to highlight the visual features and brain regions involved. However, there are two major challenges to producing a comprehensive framework for scene understanding. First, scene perception is highly dynamic, subserving multiple behavioral goals. Second, a multitude of different visual properties co-occur across scenes and may be correlated or independent. We synthesize the recent literature and argue that for a complete view of scene understanding, it is necessary to account for both differing observer goals and the contribution of diverse scene properties

Got the gist? The effects of visually evoked expectations and cross-modal stimulation on the rapid processing of real-world scenes

Scene meaning is processed rapidly, with ‘gist’ extracted even when presentation duration spans a few dozen milliseconds. This has led some to suggest a primacy of bottom-up visual information. However, gist research has typically relied on showing successions of unrelated scene images, contrary to our everyday experience of a multisensory world unfolding around us in a predictable manner. To address this lack of ecological validity, Study 1 investigated whether top-down information – in the form of observers’ predictions of an upcoming scene – facilitates gist processing. Participants (N=336) experienced a series of images, organised to represent an approach to a destination (e.g., walking down a sidewalk), followed by a final target scene either congruous or incongruous with the expected destination (e.g., a store interior or a bedroom). A series of behavioural experiments revealed that (i) appropriate expectations facilitated gist processing, (ii) inappropriate expectations interfered with gist processing, (iii) the effect of congruency was driven by provision of contextual information rather than the thematic coherence of approach images, and (iv) expectation-based facilitation was most apparent when destination duration was most curtailed. We then investigated the neural correlates of predictability on scene processing using ERP (N=26). Congruency-related differences were found in a putative scene-selective ERP component, related to integrating visual properties (P2), and in later components related to contextual integration including semantic and syntactic coherence (N400 and P600, respectively). Study 2 (N=206) then investigated the influence of simultaneous auditory information on gist processing, across two eye-tracking experiments. Search performance as a function of target sound congruency was measured using a flash-preview moving window paradigm. This revealed that a cross-modal effect did exist. Taken together, these results suggest that in real-world situations, both prior expectations and simultaneous cross-modal information influence the earliest stages of scene processing, affecting the integration of visual properties and meaning. Keywords: scene processing, gist, top-down information, event-related potentials, audio-visual processing, eye trackin