Structuring time: The hippocampus constructs sequence memories that generalize temporal relations across experiences

The hippocampal-entorhinal region supports memory for episodic details, such as temporal relations of sequential events, and mnemonic constructions combining experiences for inferential reasoning. However, it is unclear whether hippocampal event memories reflect temporal relations derived from mnemonic constructions, event order, or elapsing time, and whether these sequence representations generalize temporal relations across similar sequences. Here, participants mnemonically constructed times of events from multiple sequences using infrequent cues and their experience of passing time. After learning, event representations in the anterior hippocampus reflected temporal relations based on constructed times. Temporal relations were generalized across sequences, revealing distinct representational formats for events from the same or different sequences. Structural knowledge about time patterns, abstracted from different sequences, biased the construction of specific event times. These findings demonstrate that mnemonic construction and the generalization of relational knowledge combine in the hippocampus, consistent with the simulation of scenarios from episodic details and structural knowledge

Mnemonic construction and representation of temporal structure in the hippocampal formation

The hippocampal-entorhinal region supports memory for episodic details, such as temporal relations of sequential events, and mnemonic constructions combining experiences for inferential reasoning. However, it is unclear whether hippocampal event memories reflect temporal relations derived from mnemonic constructions, event order, or elapsing time, and whether these sequence representations generalize temporal relations across similar sequences. Here, participants mnemonically constructed times of events from multiple sequences using infrequent cues and their experience of passing time. After learning, event representations in the anterior hippocampus reflected temporal relations based on constructed times. Temporal relations were generalized across sequences, revealing distinct representational formats for events from the same or different sequences. Structural knowledge about time patterns, abstracted from different sequences, biased the construction of specific event times. These findings demonstrate that mnemonic construction and the generalization of relational knowledge combine in the hippocampus, consistent with the simulation of scenarios from episodic details and structural knowledge

Measuring long-term memories at the feature level reveals mechanisms of interference resolution

When memories share similar features, this can lead to interference, and ultimately forgetting. At the same time, many highly similar memories are remembered vividly for years to come. Understanding what causes interference and how it is overcome is key to understanding the vast human memory capacity. One unresolved challenge is that interference has primarily been studied with dichotomous measures of memory (“remembered”, “forgotten”). This limits our understanding because memories are not all-or-none, they are comprised of multiple features, each of which can be recalled with different levels of detail or bias. In order to investigate this issue, this dissertation focuses on the use of face stimuli. Faces are a unique class of stimuli for studying memory interference in that they are readily parameterizable and humans are experts at perceiving them. This means that they can be manipulated to be similar enough to cause interference, but subtle differences can also be stored and later probed from long-term memory. This dissertation develops a methodology to create synthetic faces that can be manipulated and probed along a set of perceptually-important feature dimensions. This development process included documenting face landmark positions, sorting faces based on perceived similarity, and collecting subjective ratings on a corpus of 1,148 face images. In a series of three experiments, I then applied this novel methodology to understand how memories change at the feature level when there is interference between highly similar memories. I found two memory changes that specifically occurred when there was interference between highly similar stimuli: (1) during recollection there was a bias to exaggerate the subtle differences and (2) distinguishing features were recalled with greater consistency. Critically, these memory changes were adaptive in that they were associated with less interference-related errors. Finally, in a separate fMRI experiment, I used the same corpus of faces and feature dimensions to reconstruct faces based on patterns of fMRI activity evoked while viewing them. I argue that this approach can be utilized in the future to measure neural representational changes during interference resolution. Together our findings provide important insights into how the memory system resolves interference between highly similar memories

Kognitive Interpretationen mehrdeutiger visueller Reize

Unser Gehirn muss zu jeder Zeit relevante Signale von irrelevanten Informationen trennen. Dazu müssen diese als spezifische Einheiten erkannt und klassifiziert werden. Mehrdeutigkeit ist ein wesentlicher Aspekt dieses Verarbeitungsprozesses und kann durch verrauschte Eingangssignale und durch den Aufbau unserer sensorischer Systeme entstehen. Beispielsweise können Reize mehrdeutig sein, wenn sie verrauscht oder unvollständig sind oder nur kurzzeitig wahrgenommen werden. Unter solchen Bedingungen werden Wahrnehmung und Klassifikation eines Reizes deutlich erschwert. Bereits vorhandene kognitive Repräsentationen werden somit möglicherweise nicht aktiviert. Folglich müssen Rückschlüsse über die Reize aufgrund von Kontext und Erfahrung gezogen werden. Ein und derselbe Reiz kann jedoch unterschiedlich repräsentiert und im sensorischen System kodiert werden. Da nur eine Repräsentation die Basis zukünftigen Handelns bilden kann, entsteht eine Art Konkurrenz innerhalb der Wahrnehmung. Derartige Wahrnehmungsphänomene, die mit der Mehrdeutigkeit von Reizen in Verbindung stehen, bilden den Mittelpunkt der vorliegenden Dissertation. Wenn einem physikalisch konstanten Reiz mehrere Interpretationen zugeordnet werden, entsteht ein Wechsel zwischen diesen Einordnungen, den man wahrnimmt und Rivalität ("rivalry") nennt. In dieser Dissertation werden diverse neue Erkenntnisse zu diesem grundlegenden Phänomen der sensorischen Verarbeitung beschrieben. So wird gezeigt, dass Übergänge zwischen drei wahrgenommenen Interpretationen – ein vergleichsweise selten untersuchtes Phänomen, da Rivalität meist mit zweideutigen Reizen untersucht wird – vorhersehbaren Mustern folgen (Kapitel 2). Darüber hinaus zeigt sich, dass derartige Übergänge spezifische Eigenschaften aufweisen, welche die Geschwindigkeit und die Richtung ihrer räumlichen Ausbreitung im visuellen Feld bestimmen (Kapitel 3). Diese Eigenschaften der Mehrdeutigkeit werden weiterhin stark von Aufmerksamkeit und anderen, introspektiven Prozessen beeinflusst. Um die der Rivalität in der Wahrnehmung tatsächlich zugrundeliegenden Prozesse und die damit verbundenen Änderungen des Bewusstseins von derartigen subjektiven Prozessen abzugrenzen, müssen letztere kontrolliert oder sogar vollständig umgangen werden. Ein objektives Maß der Rivalität in der Wahrnehmung wird zur Lösung dieser Aufgabe vorgeschlagen und bietet eine wertvolle Alternative zu introspektivem Berichten über den Wahrnehmungszustand (Kapitel 4). Übergänge in der Wahrnehmung entstehen entlang einer bestimmten Merkmalsdimension des Reizes, wie beispielsweise der Orientierung des berühmten Neckerwürfels. Zudem kann auch eine Änderung in der Merkmalsdimension der Luminanz eine unterschiedliche Interpretation des Reizes hervorrufen. Es wird gezeigt, dass die Pupille kleiner wird, wenn eine Interpretation mit hoher Luminanz die Wahrnehmung übernimmt, und umgekehrt, dass die Pupille größer wird, wenn eine Interpretation mit niedriger Luminanz die Wahrnehmung übernimmt. Folglich kann die Pupille als ein zuverlässiges und objektives Maß für Änderungen in der Wahrnehmung verwendet werden. Durch die Verwendung solcher objektiven Maße konnten neue Eigenschaften der Übergänge in der Wahrnehmung aufgezeigt werden, welche die Theorie unterstützen, dass Introspektion die der Verarbeitung mehrdeutiger Situationen zugrundeliegenden Prozesse merklich beeinflussen kann. Als Nächstes wurden mehrdeutiger Reize im Zusammenhang mit der Wahrnehmung von Objekten eingesetzt (Kapitel 5). Am Beispiel der Kippfigur des "bewegten Diamanten" wird dabei die Bedeutung von mehrdeutigen Reizen veranschaulicht. Beim bewegten Diamanten werden zwei Interpretationen wahrgenommen, die sich entlang der Dimension der Objektkohärenz abwechseln. Das bedeutet, dass die Wahrnehmung zwischen einem einzelnen zusammenhängenden Objekt (Diamant) und mehreren unzusammenhängenden Komponenten kippt. Es wird gezeigt, dass die Interpretation des Reizes als ein einziges kohärentes Objekt, verglichen mit der Interpretation als mehrere Komponenten, zu einer Erhöhung der visuellen Empfindlichkeit innerhalb des Objektes führt. Diese Ergebnisse sind ein Beleg dafür, wie die Aktivierung einer Interpretation eines Reizes als Einzelobjekt (im Vergleich zur Komponentenwahrnehmung) dazu führt, dass die Aufmerksamkeit top-down zu den relevanten Bereichen des Gesichtsfeldes gelenkt wird. Es wird weiter untersucht, welche Eigenschaften des Reizes zu einer bottom-up Aktivierung der Interpretation solcher Objekte beitragen (Kapitel 6). Die Mehrdeutigkeit von Objekten kann erfolgreich aufgehoben werden, indem man einen starken Kontrast in Luminanz oder Farbe zwischen dem Objekt und dem Hintergrund erzeugt. Auch die Größe und die Form haben einen großen Einfluss auf die Detektion und Identifikation von Objekten. Des Weiteren sind die Eigenschaften eines Objektes nicht nur bestimmend für die Erfolgsquote bei der Objekterkennung, sondern ebenso bedeutend für die Speicherung der Repräsentation im Gedächtnis, beispielsweise von neu wahrgenommenen Objekten. Das Klassifizieren von Objekten durch die Versuchsperson wird ebenfalls durch Mehrdeutigkeit beeinflusst. So kann ein Objekt der Versuchsperson einerseits als neu erscheinen, obwohl es bereits bekannt war, weil es beispielsweise der Versuchsperson schon einmal gezeigt worden ist. Andererseits kann auch ein eigentlich unbekanntes Objekt der Versuchsperson dennoch vertraut vorkommen. In dieser Arbeit wird gezeigt, dass solche subjektiven Effekte einen Einfluss auf die Pupillengröße haben (Kapitel 7). Außerdem verkleinert sich die Pupille der Versuchspersonen beim Betrachten neuer Bilder stärker als bei bekannten. Ein ähnlicher Effekt wird gefunden, wenn das Bild vorher erfolgreich im Gedächtnis gespeichert wurde. Daher ist es wahrscheinlich, dass die Pupille die Verfestigung von neuen Objekten im Gedächtnis widerspiegelt. Abschließend wird untersucht, ob sich kognitive Prozesse, wie Entscheidungsfindung – ein wichtiger Prozess, falls mehreren Optionen zur Verfügung stehen und Mehrdeutigkeit aufgehoben werden soll – auch in der Pupille widerspiegeln (Kapitel 8). Es wird zunächst bestätigt, dass die Pupillen sich erweitern, nachdem man eine Entscheidung getroffen hat. Neu wird gezeigt, dass diese Pupillenausdehnungen erfolgreich von anderen Personen erkannt und verwendet werden können, um ein interaktives Spiel gegen die erste Person (den "Gegner") zu gewinnen. Insgesamt wird in dieser Dissertation untersucht, wie mehrdeutige Reize die Wahrnehmung beeinflussen und wie Mehrdeutigkeit verwendet werden kann, um Prozesse des Gehirns zu studieren. Es hat sich gezeigt, dass Mehrdeutigkeit vorhersehbaren Mustern folgt, sie objektiv mit Reflexen gemessen werden kann, und Einblicke in neuronale Prozesse wie Aufmerksamkeit, Objektwahrnehmung und Entscheidungsmechanismen liefern kann. Diese Ergebnisse zeigen, dass Mehrdeutigkeit eine zentrale Eigenschaft sensorischer Systeme ist, und Lebewesen in die Lage versetzt, mit ihrer Umwelt flexibel zu interagieren. Mehrdeutigkeit macht das Verhalten vielfältiger, ermöglicht es dem Gehirn, mit der Welt auf verschiedenen Wegen zu interagieren, und ist die Basis der Dynamik von Wahrnehmung, Interpretation und Entscheidung.Brains can sense and distinguish signals from background noise in physical environments, and recognize and classify them as distinct entities. Ambiguity is an inherent part of this process. It is a cognitive property that is generated by the noisy character of the signals, and by the design of the sensory systems that process them. Stimuli can be ambiguous if they are noisy, incomplete, or only briefly sensed. Such conditions may make stimuli indistinguishable from others and thereby difficult to classify as single entities by our sensory systems. In these cases, stimuli fail to activate a representation that may have been previously stored in the system. Deduction, through context and experience, is consequently needed to reach a decision on what is exactly sensed. Deduction can, however, also be subject to ambiguity as stimuli and their properties may receive multiple representations in the sensory system. In such cases, these multiple representations compete for perceptual dominance, that is, for becoming the single entity taken by the system as a reference point for subsequent behavior. These types of ambiguity and several phenomena that relate to them are at the center of this dissertation. Perceptual rivalry, the phenomenal experience of alternating percepts over time, is an example of how the brain may give multiple interpretations to a stimulus that is physically constant. Rivalry is a very typical and general sensory process and this thesis demonstrates some newly discovered properties of its dynamics. It was found that alternations between three perceptual interpretations – a relatively rare condition as rivalry generally occurs between two percepts – follow predictable courses (Chapter 2). Furthermore, such alternations had several properties that determine their speed and direction of spatial spread (suppression waves) in the visual field (Chapter 3). These properties of ambiguity were further strongly affected by attention and other introspective processes. To demarcate the true underlying process of perceptual rivalry and the accompanied changes in awareness, these subjective processes need to be either circumvented or controlled for. An objective measure of perceptual rivalry was proposed that resolved this issue and provided a good alternative for introspective report of ambiguous states (Chapter 4). Changes in percepts occur along a specific feature domain such as depth orientation for the famous Necker cube. Alternatively, luminance may also be a rivalry feature and one percept may appear brighter as the other rivaling percept. It was demonstrated that the pupil gets smaller when a percept with high luminance becomes dominant, and vice versa, gets bigger when a percept with low luminance gets dominant during perceptual rivalry. As such, the pupil can serve as a reliable objective indicator of changes in visual awareness. By using such reflexes during rivalry, several new properties of alternations were discovered and it was again confirmed that introspection can confound the true processes involved in ambiguity. Next, the usefulness of ambiguous stimuli was explored in the context of objects as entities (Chapter 5). Some ambiguous stimuli can induce two percepts that alternate along the feature domain of object coherency, that is, whether a single coherent object or multiple incoherent objects are seen. In other words, an ambiguous stimulus can induce two cognitive interpretations of either seeing an entity or not. It was reported that being aware of a single coherent object results in the increase in visual sensitivity for the areas that constitute the object. These results are evidence of how the activation of a representation of a single and unique object can guide and allocate attentional resources to relevant areas in the visual field in a top-down way. It was further explored which features help to bottom-up access such object representations (Chapter 6). Ambiguity of objects can be successfully resolved by adding strong contrasts between the object and its background in luminance and color. The size and variability of the object's shape was also found to be an important factor for its successful detection and identification. Furthermore, the characteristics of objects do not only determine the rate of success in a recognition task, but are equally important for the storage of their representations in memory if, for instance, the object is novel to the observer. The subjective experience of a novel object is also subject to ambiguity and objects may appear novel to the observer although they are familiar (i.e., previously shown to the observer), or vice versa, they appear familiar to the observer although they are actually novel. It was here shown that such subjective effects are reflected in the pupil (Chapter 7). In addition, if novel images were presented to observers, their pupils constricted stronger as compared to if familiar images were presented. Similarly, if novel stimuli were shown to observers, pupillary constrictions were stronger if these stimuli were successfully stored in memory as compared to those later forgotten. As such, the pupil reflected the cognitive process of novelty encoding. Finally, it was tested whether other cognitive processes, such as decision-making – an important process when multiple options are available and ambiguity has to be resolved with a conscious decision – were also reflected in changes of pupil size (Chapter 8). It was confirmed that the pupil tends to dilate after an observer has made a decision. These dilations can successfully be detected between individuals and further used to gain the upper hand during an interactive game. In sum, this thesis has explored how ambiguous signals affect perception and how ambiguity inside perceptual systems can be used to study processes of the brain. It is found that ambiguity follows predictable courses, can be objectively assessed with reflexes, and can provide insights into other neuronal mechanisms such as attention, object representations, and decision-making. These findings demonstrate that ambiguity is a core property of the sensory systems that enable living beings to interact with their surroundings. Ambiguity adds variation to behavior, allows the brain to flexibly interact with the world, and lies at the bottom of the dynamics of sense, interpretations, and behavioral decisions

Schema and value: Characterizing the role of the rostral and ventral medial prefrontal cortex in episodic future thinking

As humans we are not stuck in an everlasting present. Instead, we can project ourselves into both our personal past and future. Remembering the past and simulating the future are strongly interrelated processes. They are both supported by largely the same brain regions including the rostral and ventral medial prefrontal cortex (mPFC) but also the hippocampus, the posterior cingulate cortex (PCC), as well as other regions in the parietal and temporal cortices. Interestingly, this core network for episodic simulation and episodic memory partially overlaps with a brain network for evaluation and value-based decision making. This is particularly the case for the mPFC. This part of the brain has been associated both with a large number of different cognitive functions ranging from the representation of memory schemas and self-referential processing to the representation of value and affect. As a consequence, a unifying account of mPFC functioning has remained elusive. The present thesis investigates the unique contribution of the mPFC to episodic simulation by highlighting its role in the representation of memory schemas and value. In a first functional MRI and pre-registered behavioral replication study, we demonstrate that the mPFC encodes representations of known people as well as of known locations from participants’ everyday life. We demonstrate that merely imagined encounters with liked vs. disliked people at these locations can change our attitude toward the locations. The magnitude of this simulation-induced attitude change was predicted by activation in the mPFC during the simulations. Specifically, locations simulated with liked people exhibited significantly larger increases in liking than those simulated with disliked people. In a second behavioral study, we examined the mechanisms of simulation-based learning more closely. To this end, participants also simulated encounters with neutral people at neutral locations. Using repeated behavioral assessments of participants’ memory representations, we reveal that simulations cause an integration of memory representations for jointly simulated people and locations. Moreover, compared to the neutral baseline condition we demonstrate a transfer of positive valence from liked and of negative valence from disliked people to their paired locations. We also provide evidence that simulations induce an affective experience that aligns with the valence of the person and that this experience can account for the observed attitude change toward the location. In a final fMRI study, we examine the structure of memory representations encoded in the mPFC. Specifically, we provide evidence for the hypothesis that the mPFC encodes schematic representations of our social and physical environment. We demonstrate that representations of individual exemplars of these environments (i.e., individual people and locations) are closely intertwined with a representation of their value. In sum, our findings show that we can learn from imagined experience much as we learn from actual past experience and that the mPFC plays a key role in simulation-based learning. The mPFC encodes information about our environment in value-weighted schematic representations. These representations can account for the overlap of mnemonic and evaluative functions in the mPFC and might play a key role in simulation-based learning. Our results are in line with a view that our memories of the past serve us in ways that are oriented toward the future. Our ability to simulate potential scenarios allows us to anticipate the future consequences of our choices and thereby fosters farsighted decision making. Thus, our findings help to better characterize the functional role of the mPFC in episodic future simulation and valuation

Latent Factor Analysis of High-Dimensional Brain Imaging Data

Recent advances in neuroimaging study, especially functional magnetic resonance imaging (fMRI), has become an important tool in understanding the human brain. Human cognitive functions can be mapped with the brain functional organization through the high-resolution fMRI scans. However, the high-dimensional data with the increasing number of scanning tasks and subjects pose a challenge to existing methods that wasn’t optimized for high-dimensional imaging data. In this thesis, I develop advanced data-driven methods to help utilize more available sources of information in order to reveal more robust brain-behavior relationship. In the first chapter, I provide an overview of the current related research in fMRI and my contributions to the field. In the second chapter, I propose two extensions to the connectome-based predictive modeling (CPM) method that is able to combine multiple connectomes when building predictive models. The two extensions are both able to generate higher prediction accuracy than using the single connectome or the average of multiple connectomes, suggesting the advantage of incorporating multiple sources of information in predictive modeling. In the third chapter, I improve CPM from the target behavioral measure’s perspective. I propose another two extensions for CPM that are able to combine multiple available behavioral measures into a composite measure for CPM to predict. The derived composite measures are shown to be predicted more accurately than any other single behavioral measure, suggesting a more robust brainbehavior relationship. In the fourth chapter, I propose a nonlinear dimensionality reduction framework to embed fMRI data from multiple tasks into a low-dimensional space. This framework helps reveal the common brain state in the multiple available tasks while also help discover the differences among these tasks. The results also provide valuable insights into the various prediction performance based on connectomes from different tasks. In the fifth chapter, I propose an another hyerbolic geometry-based brain graph edge embedding framework. The framework is based on Poincar´e embedding and is able to more accurately represent edges in the brain graph in a low-dimensional space than traditional Euclidean geometry-based embedding. Utilizing the embedding, we are able to cluster edges of the brain graph into disjoint clusters. The edge clusters can then be used to define overlapping brain networks and the derived metrics like network overlapping number can be used to investigate functional flexibility of each brain region. Overall, these work provide rich data-driven methods that help understand the brain-behavioral relationship through predictive modeling and low-dimensional data representation

The Unconscious Formation of Motor and Abstract Intentions

Three separate fMRI studies were conducted to study the neural dynamics of free decision formation. In Study 1, we first searched across the brain for spatiotemporal patterns that could predict the specific outcome and timing of free motor decisions to make a left or right button press (Soon et al., 2008). In Study 2, we replicated Study 1 using ultra-high field fMRI for improved temporal and spatial resolution to more accurately characterize the evolution of decision-predictive information in prefrontal cortex (Bode et al., 2011). In Study 3, to unequivocally dissociate high-level intentions from motor preparation and execution, we investigated the neural precursors of abstract intentions as participants spontaneously decided to perform either of two mental arithmetic tasks: addition or subtraction (Soon et al., 2013). Across the three studies, we consistently found that upcoming decisions could be predicted with ~60% accuracy from fine-grained spatial activation patterns occurring a few seconds before the decisions reached awareness, with very similar profiles for both motor and abstract intentions. The content and timing of the decisions appeared to be encoded in two functionally dissociable sets of regions: frontopolar and posterior cingulate/ precuneus cortex encoded the content but not the timing of the decisions, while the pre-supplementary motor area encoded the timing but not the content of the decisions. The choice-predictive regions in both motor and abstract decision tasks overlapped partially with the default mode network. High-resolution imaging in Study 2 further revealed that as the time-point of conscious decision approached, activity patterns in frontopolar cortex became increasingly stable with respect to the final choice.:Abstract 1 1. General Introduction 5 2. Study 1: Decoding the Unconscious Formation of Motor Intentions 21 3. Study 2: Temporal Stability of Neural Patterns Involved in Intention Formation 56 4. Study 3: Decoding the Unconscious Formation of Abstract Intentions 89 5. General Discussion 119 References 14