Episodic visual cognition: Implications for object and short-term recognition

Poth CH. Episodic visual cognition: Implications for object and short-term recognition. Bielefeld: Universität Bielefeld; 2017.A great part of human behavior is based on visual cognition, the processing of visual information about external objects. For goal-directed behavior, two functions of visual cognition seem especially important. The first one is object recognition. Objects in the environment must be identified as belonging to an object category, so that they can be used to accomplish a given task. The second function is short-term recognition. It must be recognized whether an object in the environment has been viewed recently, so that current behavior involving the object can be related to previous behavior. Both functions share a common constraint: They must be fulfilled across distinct episodes of visual processing, which are interrupted by changes in processing demands. For object recognition, visual processing episodes lead to a problem of selective integration. That is, it must be decided whether object information from the current episode should update and thus be integrated with object representations from the previous episode. Alternatively, object representations from two successive episodes are retained separately. This decision is critical. Updating and integration should enable a cumulative and fast object recognition. However, integration should also conceal object changes across episodes by leaving no separate representations that can be compared. Separation should improve change perception but impair object recognition, because limited visual processing resources for object recognition must be split between the separate representations. For short-term recognition visual processing episodes lead to a problem of matching. That is, an object from the current episode must be matched against object representations, not only from the previous but from several episodes in the recent past (irrespective of whether objects are categorized). The overarching goal of the present dissertation is to make a first step in understanding how the mechanisms underlying object recognition and short-term recognition operate across visual processing episodes, and how they solve the two problems. In five empirical studies, we investigated key issues that must be addressed before a theoretical account of object and short-term recognition across visual processing episodes can be given. The first three studies focused on object recognition across visual processing episodes of eye fixations. Fixations are periods of visual information uptake, in which the eyes stand relatively still. They are separated by rapid saccadic eye movements. Saccades are necessary for object recognition, because they direct the central fovea of the eye's retina at interesting objects, allowing high-acuity inspection. However, saccades also disrupt visual input and displace and alter the retinal images of objects. Therefore, saccades dissect visual information processing into distinct episodes of fixations, which the mechanisms for object recognition must accommodate. In two studies (Poth, Herwig, & Schneider, 2015; Poth & Schneider, 2016a), we investigated how the selective integration problem is solved to support object recognition across successive fixations. We assessed the recent hypothesis (Schneider, 2013) that the problem is solved by a mechanism testing for correspondence (``object continuity'') between an object before and after a saccade. If object correspondence is established, the object before and after the saccade should be integrated into a common representation. In contrast, if object correspondence is broken, the object before and after the saccade should be represented separately. Separation should allow to compare the two representations, improving the discrimination of transsaccadic object displacements. At the same time, however, object recognition of the object after the saccade should be impaired, because the necessary visual processing resources had to be split between the two representations. Results were consistent with this hypothesis. Breaking object correspondence by briefly blanking an object after a saccade to it improved the discrimination of displacements of the object but impaired object recognition. Thus, the object correspondence mechanism seems to impact on object recognition after the saccade. Further experiments investigated the nature of object correspondence. They showed that object recognition was impaired when object correspondence was broken by changing an object's contrast-polarity (and luminance), its color-and-luminance, and its color alone. Together with the initial finding, this indicates that object correspondence is based on spatiotemporal as well as on the surface features of objects. In the third study (Poth & Schneider, 2016b, submitted), we went on to test the limits of object recognition across saccades. Because object recognition relies on limited visual processing resources, it can only be achieved for a few objects at a time. Here, we examined if different objects must compete for these resources across saccades. If this was the case, visual processing after a saccade would be slowed down as more and more objects are viewed before the saccade. Our findings show that this is the case, but only if the objects are task-relevant. Therefore, the findings support a key prediction of a recent theory, namely that the importance of an object representation determines whether it will survive a saccade and take up limited processing resources afterwards (Schneider, 2013). With the fourth study (Poth & Schneider, 2016c), we turned from the processing episodes of successive eye fixations to those defined by appearing and disappearing objects and associated task-requirements. We asked about the relationship between the mechanisms underlying object recognition and those underlying short-term recognition. Visual processing for object recognition is assumed to be complete when an object has entered a limited-capacity visual working memory, where the object becomes available for being reported. We investigated if encoding into visual working memory is not only required for object recognition in the current episode, but also for short-term recognition in upcoming episodes. Supporting this notion, we found that objects that supposedly had not reached visual working memory were not available for later short-term recognition. This finding argues that the initial steps of visual processing before encoding into visual working memory are not sufficient for short-term recognition in later episodes. Therefore, visual working memory may contribute to the solution of the matching problem by limiting the amount of information considered in a short-term recognition task. Finally, in the fifth study (Poth & Schneider, 2016d, submitted), we investigated short-term recognition further, asking how short-term recognition in a later processing episode can be prepared in advance. We assessed how prioritizing among objects represented in visual working memory impacts on two distinct components of performance in an upcoming short-term recognition task. Our results showed that such a prioritization improves memory-retention in visual working memory but also accelerates visual processing of objects for short-term recognition in a future episode. This indicates that changes in processing priorities contribute to ongoing solutions of the matching problem of short-term recognition. Taken together, the five studies show how mechanisms of object and short-term recognition address specific problems arising from the dissection of visual processing into distinct episodes. As such, the studies implicate visual processing episodes as a source of problems for object and short-term recognition, which is neglected in most contemporary research. Conversely, however, the studies also invite speculation about the functional value of visual processing episodes for visual cognition

Episodic Short-Term Recognition Requires Encoding into Visual Working Memory: Evidence from Probe Recognition after Letter Report

Poth CH, Schneider WX. Episodic Short-Term Recognition Requires Encoding into Visual Working Memory: Evidence from Probe Recognition after Letter Report. Frontiers in Psychology. 2016;7: 1440.Human vision is organized in discrete processing episodes (e.g., eye fixations or task-steps). Object information must be transmitted across episodes to enable episodic short-term recognition: recognizing whether a current object has been seen in a previous episode. We ask whether episodic short-term recognition presupposes that objects have been encoded into capacity-limited visual working memory (VWM), which retains visual information for report. Alternatively, it could rely on the activation of visual features or categories that occurs before encoding into VWM. We assessed the dependence of episodic short-term recognition on VWM by a new paradigm combining letter report and probe recognition. Participants viewed displays of 10 letters and reported as many as possible after a retention interval (whole report). Next, participants viewed a probe letter and indicated whether it had been one of the 10 letters (probe recognition). In Experiment 1, probe recognition was more accurate for letters that had been encoded into VWM (reported letters) compared with non-encoded letters (non-reported letters). Interestingly, those letters that participants reported in their whole report had been near to one another within the letter displays. This suggests that the encoding into VWM proceeded in a spatially clustered manner. In Experiment 2, participants reported only one of 10 letters (partial report) and probes either referred to this letter, to letters that had been near to it, or far from it. Probe recognition was more accurate for near than for far letters, although none of these letters had to be reported. These findings indicate that episodic short-term recognition is constrained to a small number of simultaneously presented objects that have been encoded into VWM

Breaking object correspondence across saccadic eye movements deteriorates object recognition

Poth CH, Herwig A, Schneider WX. Breaking object correspondence across saccadic eye movements deteriorates object recognition. Frontiers in Systems Neuroscience. 2015;9: 176.Visual perception is based on information processing during periods of eye fixations that are interrupted by fast saccadic eye movements. The ability to sample and relate information on task-relevant objects across fixations implies that correspondence between presaccadic and postsaccadic objects is established. Postsaccadic object information usually updates and overwrites information on the corresponding presaccadic object. The presaccadic object representation is then lost. In contrast, the presaccadic object is conserved when object correspondence is broken. This helps transsaccadic memory but it may impose attentional costs on object recognition. Therefore, we investigated how breaking object correspondence across the saccade affects postsaccadic object recognition. In Experiment 1, object correspondence was broken by a brief postsaccadic blank screen. Observers made a saccade to a peripheral object which was displaced during the saccade. This object reappeared either immediately after the saccade or after the blank screen. Within the postsaccadic object, a letter was briefly presented (terminated by a mask). Observers reported displacement direction and letter identity in different blocks. Breaking object correspondence by blanking improved displacement identification but deteriorated postsaccadic letter recognition. In Experiment 2, object correspondence was broken by changing the object’s contrast-polarity. There were no object displacements and observers only reported letter identity. Again, breaking object correspondence deteriorated postsaccadic letter recognition. These findings identify transsaccadic object correspondence as a key determinant of object recognition across the saccade. This is in line with the recent hypothesis that breaking object correspondence results in separate representations of presaccadic and postsaccadic objects which then compete for limited attentional processing resources (Schneider, 2013). Postsaccadic object recognition is then deteriorated because less resources are available for processing postsaccadic objects

Effects of monitoring for visual events on distinct components of attention

Poth CH, Petersen A, Bundesen C, Schneider WX. Effects of monitoring for visual events on distinct components of attention. Frontiers in Psychology. 2014;5:930.Monitoring the environment for visual events while performing a concurrent task requires adjustment of visual processing priorities. By use of Bundesen’s (1990) Theory of Visual Attention, we investigated how monitoring for an object-based brief event affected distinct components of visual attention in a concurrent task. The perceptual salience of the event was varied. Monitoring reduced the processing speed in the concurrent task, and the reduction was stronger when the event was less salient. The monitoring task neither affected the temporal threshold of conscious perception nor the storage capacity of visual short-term memory nor the efficiency of top-down controlled attentional selection

Using the virtual reality device Oculus Rift for neuropsychological assessment of visual processing capabilities

Foerster RM, Poth CH, Behler C, Botsch M, Schneider WX. Using the virtual reality device Oculus Rift for neuropsychological assessment of visual processing capabilities. Scientific Reports. 2016;6(1): 37016.Neuropsychological assessment of human visual processing capabilities strongly depends on visual testing conditions including room lighting, stimuli, and viewing-distance. This limits standardization, threatens reliability, and prevents the assessment of core visual functions such as visual processing speed. Increasingly available virtual reality devices allow to address these problems. One such device is the portable, light-weight, and easy-to-use Oculus Rift. It is head-mounted and covers the entire visual field, thereby shielding and standardizing the visual stimulation. A fundamental prerequisite to use Oculus Rift for neuropsychological assessment is sufficient test-retest reliability. Here, we compare the test-retest reliabilities of Bundesen’s visual processing components (visual processing speed, threshold of conscious perception, capacity of visual working memory) as measured with Oculus Rift and a standard CRT computer screen. Our results show that Oculus Rift allows to measure the processing components as reliably as the standard CRT. This means that Oculus Rift is applicable for standardized and reliable assessment and diagnosis of elementary cognitive functions in laboratory and clinical settings. Oculus Rift thus provides the opportunity to compare visual processing components between individuals and institutions and to establish statistical norm distributions

Ultrahigh temporal resolution of visual presentation using gaming monitors and G-Sync

Poth CH, Foerster RM, Behler C, Schwanecke U, Schneider WX, Botsch M. Ultrahigh temporal resolution of visual presentation using gaming monitors and G-Sync. Behavior Research Methods. 2018;50(1):26-38.all implementations, we confirmed the ultra-high temporal resolution of visual presentation with external measurements using a photodiode. Moreover, a psychophysical experiment revealed that the ultra-high temporal resolution impacts on human visual performance. Specifically, observers’ object recognition performance improved over fine-grained increases of object presentation duration in a theoretically predicted way. Taken together, the present study shows that the G- Sync-based presentation method enables to investigate visual processes whose data patterns were concealed by the low temporal resolution of previous technologies. Therefore, this new presentation method may be a valuable tool for experimental psychologists and neuroscientists studying vision and its temporal characteristics

Paying attention

Poth CH. Paying attention. eLife. 2024;13: e99560.Intentional eye movements depend on where peripheral attention is voluntarily deployed beforehand, but they can be directed elsewhere shortly afterwards. © 2024, Poth

Project "Phasic alertness and accessory stimulation" (AACRT)

The project investigates links between phasic alertness and accessory stimulation in a visual choice reaction task

Urgency forces stimulus-driven action by overcoming cognitive control

Poth CH. Urgency forces stimulus-driven action by overcoming cognitive control. eLife. 2021;10: e73682.Intelligent behavior requires to act directed by goals despite competing action tendencies triggered by stimuli in the environment. For eye movements, it has recently been discovered that this ability is briefly reduced in urgent situations (Salinas et al., 2019). In a time-window before an urgent response, participants could not help but look at a suddenly appearing visual stimulus, even though their goal was to look away from it. Urgency seemed to provoke a new visual-oculomotor phenomenon: A period in which saccadic eye movements are dominated by external stimuli, and uncontrollable by current goals. This period was assumed to arise from brain mechanisms controlling eye movements and spatial attention, such as those of the frontal eye field. Here, we show that the phenomenon is more general than previously thought. We found that also in well-investigated manual tasks, urgency made goal-conflicting stimulus features dominate behavioral responses. This dominance of behavior followed established trial-to-trial signatures of cognitive control mechanisms that replicate across a variety of tasks. Thus together, these findings reveal that urgency temporarily forces stimulus-driven action by overcoming cognitive control in general, not only at brain mechanisms controlling eye movements. © 2021, Poth