Extracting scene and object information from natural stimuli: the influence of scene structure and eye movements

When we observe a scene in our daily lives, our brains seemingly effortlessly extract various aspects of that scene. This can be attributed to different aspects of the human visual system, including but not limited to (1) its tuning to natural regularities in scenes and (2) its ability to bring different parts of the visual environment into focus via eye movements. While eye movements are a ubiquitous and natural behavior, they are considered undesirable in many highly controlled visual experiments. Participants are often instructed to fixate but cannot always suppress involuntary eye movements, which can challenge the interpretation of neuroscientific data, in particular for magneto- and electroencephalography (M/EEG). This dissertation addressed how scene structure and involuntary eye movements influence the extraction of scene and object information from natural stimuli. First, we investigated when and where real-world scene structure affects scene-selective cortical responses. Second, we investigated whether spatial structure facilitates the temporal analysis of a scene’s categorical content. Third, we investigated whether the spatial content of a scene aids in extracting task-relevant object information. Fourth, we explored whether the choice of fixation cross influences eye movements and the classification of natural images from EEG and eye tracking. The first project showed that spatial scene structure impacts scene-selective neural responses in OPA and PPA, revealing genuine sensitivity to spatial scene structure starting from 255 ms, while scene-selective neural responses are less sensitive to categorical scene structure. The second project demonstrated that spatial scene structure facilitates the extraction of the scene’s categorical content within 200 ms of vision. The third project showed that coherent scene structure facilitates the extraction of object information if the object is task-relevant, suggesting a task-based modulation. The fourth project showed that choosing a centrally presented bullseye instead of a standard fixation cross reduces eye movements on the single image level and subtly removes systematic eye movement related activity in M/EEG data. Taken together, the results advanced our understanding of (1) the impact of real-world structure on scene perception as well as the extraction of object information and (2) the influence of eye movements on advanced analysis methods.Wenn wir in unserem täglichen Leben eine Szene beobachten, extrahiert unser Gehirn scheinbar mühelos verschiedene Aspekte dieser Szene. Dies kann auf verschiedene Aspekte des menschlichen Sehsystems zurückgeführt werden, unter anderem auf (1) seine Ausrichtung auf natürliche Regelmäßigkeiten in Szenen und (2) seine Fähigkeit, verschiedene Teile der visuellen Umgebung durch Augenbewegungen in den Fokus zu bringen. Obwohl Augenbewegungen ein allgegenwärtiges und natürliches Verhalten sind, werden sie in vielen stark kontrollierten visuellen Experimenten als unerwünscht angesehen. Die Teilnehmer werden oft angewiesen, zu fixieren, können aber unwillkürliche Augenbewegungen nicht immer unterdrücken, was die Interpretation neurowissenschaftlicher Daten, insbesondere der Magneto- und Elektroenzephalographie (M/EEG), in Frage stellen kann. In dieser Dissertation wurde untersucht, wie Szenenstruktur und unbewusste Augenbewegungen die Extraktion von Szenen- und Objektinformationen aus natürlichen Stimuli beeinflussen. Zunächst untersuchten wir, wann und wo die Struktur einer realen Szene die szenenselektiven kortikalen Reaktionen beeinflusst. Zweitens untersuchten wir, ob die räumliche Struktur die zeitliche Analyse des kategorialen Inhalts einer Szene erleichtert. Drittens untersuchten wir, ob der räumliche Inhalt einer Szene bei der Extraktion aufgabenrelevanter Objektinformationen hilft. Viertens untersuchten wir, ob die Wahl des Fixationskreuzes die Augenbewegungen und die Klassifizierung natürlicher Bilder aus EEG und Eye-Tracking beeinflusst. Das erste Projekt zeigte, dass sich die räumliche Szenenstruktur auf szenenselektive neuronale Reaktionen in OPA und PPA auswirkt, wobei eine echte Empfindlichkeit für räumliche Szenenstrukturen ab 255 ms festgestellt wurde, während szenenselektive neuronale Reaktionen weniger empfindlich auf kategoriale Szenenstrukturen reagieren. Das zweite Projekt zeigte, dass die räumliche Szenenstruktur die Extraktion des kategorialen Inhalts der Szene innerhalb von 200 ms nach dem Sehen erleichtert. Das dritte Projekt zeigte, dass eine kohärente Szenenstruktur die Extraktion von Objektinformationen erleichtert, wenn das Objekt aufgabenrelevant ist, was auf eine aufgabenbezogene Modulation hindeutet. Das vierte Projekt zeigte, dass die Wahl eines zentral präsentierten Bullauges anstelle eines Standard-Fixationskreuzes Augenbewegungen auf Einzelbildebene reduziert und systematische Augenbewegungsaktivität in M/EEG-Daten auf subtile Weise beseitigt. Zusammengenommen haben die Ergebnisse unser Verständnis (1) der Auswirkungen der Struktur der realen Welt auf die Wahrnehmung der Szene und die Extraktion von Objektinformationen und (2) des Einflusses von Augenbewegungen auf fortgeschrittene Analysemethoden verbessert

Single Neuron Correlates of Learning, Value, and Decision in the Human Brain

In this thesis, I present several new results on how the human brain performs value-based learning and decision-making, leveraging rare single neuron recordings from epilepsy patients in vmPFC, preSMA, dACC, amygdala, and hippocampus, as well as reinforcement learning models of behavior. With a probabilistic gambling task we determined that human preSMA neurons integrate computational components of stimulus value such as expected values, uncertainty, and novelty, to encode an utility value and, subsequently, decisions themselves. Additionally, we found that post-decision related encoding of variables for the chosen option was more widely distributed and especially prominent in vmPFC. Additionally, with a Pavlovian conditioning task we found evidence of stimulus-stimulus associations in vmPFC, while both vmPFC and amygdala performed predictive value coding, establishing direct evidence for model-based Pavlovian conditioning in human vmPFC neurons. Finally, in a Pavlovian observational learning paradigm, we found a significant proportion of amygdala neurons whose activity correlated with both expected rewards for oneself and others, and in tracking outcome values received by oneself or other agents, further establishing amygdala as an important center in social cognition. Taken together, our findings expand our understanding of the role of several human cortical brain regions in creating and updating value representations which are leveraged during decision-making.</p

The role of phonology in visual word recognition: evidence from Chinese

Posters - Letter/Word Processing V: abstract no. 5024The hypothesis of bidirectional coupling of orthography and phonology predicts that phonology plays a role in visual word recognition, as observed in the effects of feedforward and feedback spelling to sound consistency on lexical decision. However, because orthography and phonology are closely related in alphabetic languages (homophones in alphabetic languages are usually orthographically similar), it is difficult to exclude an influence of orthography on phonological effects in visual word recognition. Chinese languages contain many written homophones that are orthographically dissimilar, allowing a test of the claim that phonological effects can be independent of orthographic similarity. We report a study of visual word recognition in Chinese based on a mega-analysis of lexical decision performance with 500 characters. The results from multiple regression analyses, after controlling for orthographic frequency, stroke number, and radical frequency, showed main effects of feedforward and feedback consistency, as well as interactions between these variables and phonological frequency and number of homophones. Implications of these results for resonance models of visual word recognition are discussed.postprin

Interactive effects of orthography and semantics in Chinese picture naming

Posters - Language Production/Writing: abstract no. 4035Picture-naming performance in English and Dutch is enhanced by presentation of a word that is similar in form to the picture name. However, it is unclear whether facilitation has an orthographic or a phonological locus. We investigated the loci of the facilitation effect in Cantonese Chinese speakers by manipulating—at three SOAs (2100, 0, and 1100 msec)—semantic, orthographic, and phonological similarity. We identified an effect of orthographic facilitation that was independent of and larger than phonological facilitation across all SOAs. Semantic interference was also found at SOAs of 2100 and 0 msec. Critically, an interaction of semantics and orthography was observed at an SOA of 1100 msec. This interaction suggests that independent effects of orthographic facilitation on picture naming are located either at the level of semantic processing or at the lemma level and are not due to the activation of picture name segments at the level of phonological retrieval.postprin

How to improve learning from video, using an eye tracker

The initial trigger of this research about learning from video was the availability of log files from users of video material. Video modality is seen as attractive as it is associated with the relaxed mood of watching TV. The experiments in this research have the goal to gain more insight in viewing patterns of students when viewing video. Students received an awareness instruction about the use of possible alternative viewing behaviors to see whether this would enhance their learning effects. We found that: - the learning effects of students with a narrow viewing repertoire were less than the learning effects of students with a broad viewing repertoire or strategic viewers. - students with some basic knowledge of the topics covered in the videos benefited most from the use of possible alternative viewing behaviors and students with low prior knowledge benefited the least. - the knowledge gain of students with low prior knowledge disappeared after a few weeks; knowledge construction seems worse when doing two things at the same time. - media players could offer more options to help students with their search for the content they want to view again. - there was no correlation between pervasive personality traits and viewing behavior of students. The right use of video in higher education will lead to students and teachers that are more aware of their learning and teaching behavior, to better videos, to enhanced media players, and, finally, to higher learning effects that let users improve their learning from video

Brain Computations and Connectivity [2nd edition]

This is an open access title available under the terms of a CC BY-NC-ND 4.0 International licence. It is free to read on the Oxford Academic platform and offered as a free PDF download from OUP and selected open access locations. Brain Computations and Connectivity is about how the brain works. In order to understand this, it is essential to know what is computed by different brain systems; and how the computations are performed. The aim of this book is to elucidate what is computed in different brain systems; and to describe current biologically plausible computational approaches and models of how each of these brain systems computes. Understanding the brain in this way has enormous potential for understanding ourselves better in health and in disease. Potential applications of this understanding are to the treatment of the brain in disease; and to artificial intelligence which will benefit from knowledge of how the brain performs many of its extraordinarily impressive functions. This book is pioneering in taking this approach to brain function: to consider what is computed by many of our brain systems; and how it is computed, and updates by much new evidence including the connectivity of the human brain the earlier book: Rolls (2021) Brain Computations: What and How, Oxford University Press. Brain Computations and Connectivity will be of interest to all scientists interested in brain function and how the brain works, whether they are from neuroscience, or from medical sciences including neurology and psychiatry, or from the area of computational science including machine learning and artificial intelligence, or from areas such as theoretical physics

Programming the cerebellum

It is argued that large-scale neural network simulations of cerebellar cortex and nuclei, based on realistic compartmental models of me major cell populations, are necessary before the problem of motor learning in the cerebellum can be solved, [HOUK et al.; SIMPSON et al.