The Construction of Semantic Memory: Grammar-Based Representations Learned from Relational Episodic Information

After acquisition, memories underlie a process of consolidation, making them more resistant to interference and brain injury. Memory consolidation involves systems-level interactions, most importantly between the hippocampus and associated structures, which takes part in the initial encoding of memory, and the neocortex, which supports long-term storage. This dichotomy parallels the contrast between episodic memory (tied to the hippocampal formation), collecting an autobiographical stream of experiences, and semantic memory, a repertoire of facts and statistical regularities about the world, involving the neocortex at large. Experimental evidence points to a gradual transformation of memories, following encoding, from an episodic to a semantic character. This may require an exchange of information between different memory modules during inactive periods. We propose a theory for such interactions and for the formation of semantic memory, in which episodic memory is encoded as relational data. Semantic memory is modeled as a modified stochastic grammar, which learns to parse episodic configurations expressed as an association matrix. The grammar produces tree-like representations of episodes, describing the relationships between its main constituents at multiple levels of categorization, based on its current knowledge of world regularities. These regularities are learned by the grammar from episodic memory information, through an expectation-maximization procedure, analogous to the inside–outside algorithm for stochastic context-free grammars. We propose that a Monte-Carlo sampling version of this algorithm can be mapped on the dynamics of “sleep replay” of previously acquired information in the hippocampus and neocortex. We propose that the model can reproduce several properties of semantic memory such as decontextualization, top-down processing, and creation of schemata

Sensory Processing Across Conscious and Nonconscious Brain States: From Single Neurons to Distributed Networks for Inferential Representation

Neuronal activity is markedly different across brain states: it varies from desynchronized activity during wakefulness to the synchronous alternation between active and silent states characteristic of deep sleep. Surprisingly, limited attention has been paid to investigating how brain states affect sensory processing. While it was long assumed that the brain was mostly disconnected from external stimuli during sleep, an increasing number of studies indicates that sensory stimuli continue to be processed across all brain states—albeit differently. In this review article, we first discuss what constitutes a brain state. We argue that—next to global, behavioral states such as wakefulness and sleep—there is a concomitant need to distinguish bouts of oscillatory dynamics with specific global/local activity patterns and lasting for a few hundreds of milliseconds, as these can lead to the same sensory stimulus being either perceived or not. We define these short-lasting bouts as micro-states. We proceed to characterize how sensory-evoked neural responses vary between conscious and nonconscious states. We focus on two complementary aspects: neuronal ensembles and inter-areal communication. First, we review which features of ensemble activity are conducive to perception, and how these features vary across brain states. Properties such as heterogeneity, sparsity and synchronicity in neuronal ensembles will especially be considered as essential correlates of conscious processing. Second, we discuss how inter-areal communication varies across brain states and how this may affect brain operations and sensory processing. Finally, we discuss predictive coding (PC) and the concept of multi-level representations as a key framework for understanding conscious sensory processing. In this framework the brain implements conscious representations as inferences about world states across multiple representational levels. In this representational hierarchy, low-level inference may be carried out nonconsciously, whereas high levels integrate across different sensory modalities and larger spatial scales, correlating with conscious processing. This inferential framework is used to interpret several cellular and population-level findings in the context of brain states, and we briefly compare its implications to two other theories of consciousness. In conclusion, this review article, provides foundations to guide future studies aiming to uncover the mechanisms of sensory processing and perception across brain states

Deep Gated Hebbian Predictive Coding Accounts for Emergence of Complex Neural Response Properties Along the Visual Cortical Hierarchy

Predictive coding provides a computational paradigm for modeling perceptual processing as the construction of representations accounting for causes of sensory inputs. Here, we developed a scalable, deep network architecture for predictive coding that is trained using a gated Hebbian learning rule and mimics the feedforward and feedback connectivity of the cortex. After training on image datasets, the models formed latent representations in higher areas that allowed reconstruction of the original images. We analyzed low- and high-level properties such as orientation selectivity, object selectivity and sparseness of neuronal populations in the model. As reported experimentally, image selectivity increased systematically across ascending areas in the model hierarchy. Depending on the strength of regularization factors, sparseness also increased from lower to higher areas. The results suggest a rationale as to why experimental results on sparseness across the cortical hierarchy have been inconsistent. Finally, representations for different object classes became more distinguishable from lower to higher areas. Thus, deep neural networks trained using a gated Hebbian formulation of predictive coding can reproduce several properties associated with neuronal responses along the visual cortical hierarchy

Local minimization of prediction errors drives learning of invariant object representations in a generative network model of visual perception

The ventral visual processing hierarchy of the cortex needs to fulfill at least two key functions: perceived objects must be mapped to high-level representations invariantly of the precise viewing conditions, and a generative model must be learned that allows, for instance, to fill in occluded information guided by visual experience. Here, we show how a multilayered predictive coding network can learn to recognize objects from the bottom up and to generate specific representations via a top-down pathway through a single learning rule: the local minimization of prediction errors. Trained on sequences of continuously transformed objects, neurons in the highest network area become tuned to object identity invariant of precise position, comparable to inferotemporal neurons in macaques. Drawing on this, the dynamic properties of invariant object representations reproduce experimentally observed hierarchies of timescales from low to high levels of the ventral processing stream. The predicted faster decorrelation of error-neuron activity compared to representation neurons is of relevance for the experimental search for neural correlates of prediction errors. Lastly, the generative capacity of the network is confirmed by reconstructing specific object images, robust to partial occlusion of the inputs. By learning invariance from temporal continuity within a generative model, the approach generalizes the predictive coding framework to dynamic inputs in a more biologically plausible way than self-supervised networks with non-local error-backpropagation. This was achieved simply by shifting the training paradigm to dynamic inputs, with little change in architecture and learning rule from static input-reconstructing Hebbian predictive coding networks

Consciousness Regained: Disentangling Mechanisms, Brain Systems, and Behavioral Responses

How consciousness (experience) arises from and relates to material brain processes (the "mind-body problem") has been pondered by thinkers for centuries, and is regarded as among the deepest unsolved problems in science, with wide-ranging theoretical, clinical, and ethical implications. Until the last few decades, this was largely seen as a philosophical topic, but not widely accepted in mainstream neuroscience. Since the 1980s, however, novel methods and theoretical advances have yielded remarkable results, opening up the field for scientific and clinical progress. Since a seminal paper by Crick and Koch (1998) claimed that a science of consciousness should first search for its neural correlates (NCC), a variety of correlates have been suggested, including both content-specific NCCs, determining particular phenomenal components within an experience, and the full NCC, the neural substrates supporting entire conscious experiences. In this review, we present recent progress on theoretical, experimental, and clinical issues. Specifically, we (1) review methodological advances that are important for dissociating conscious experience from related enabling and executive functions, (2) suggest how critically reconsidering the role of the frontal cortex may further delineate NCCs, (3) advocate the need for general, objective, brain-based measures of the capacity for consciousness that are independent of sensory processing and executive functions, and (4) show how animal studies can reveal population and network phenomena of relevance for understanding mechanisms of consciousness.European Union's Horizon 2020 Research and Innovation ProgrammeHermann and Lilly Schilling FoundationGerman Research FoundationCenter for Nanoscale Microscopy and Molecular Physiology of the BrainNational Institutes of Health/National Institute of Neurological Disorders and StrokeSao Paulo Research FoundationJames S. McDonnell Foundation Scholar AwardEU Grant H2020-FETOPENCanadian Institute for Advanced ResearchAzrieli Program in Brain, Mind and ConsciousnessFLAG-ERA JTC project CANONNorwegian Research CouncilNetherlands Organization for Scientific ResearchUniv Oslo, Inst Basal Med Sci, Div Physiol, Dept Mol Med, POB 1103 Blindern, N-0317 Oslo, NorwayUniv Wisconsin, Dept Neurol, Madison, WI 53705 USAUniv Wisconsin, Dept Psychiat, Madison, WI 53719 USAUniv Fed Sao Paulo, Inst Sci & Technol, BR-12231280 Sao Jose Dos Campos, SP, BrazilUniv Milan, Dept Biomed & Clin Sci Luigi Sacco, I-20157 Milan, ItalyFdn Don Carlo Gnocchi ONLUS, Ist Ricovero & Cura Carattere Sci, I-20162 Milan, ItalyUniv Amsterdam, Swammerdam Inst Life Sci, Cognit & Syst Neurosci Grp, NL-1098 XH Amsterdam, NetherlandsUniv Amsterdam, Res Prior Program Brain & Cognit, NL-1098 XH Amsterdam, NetherlandsUniv Med Goettingen, Dept Cognit Neurol, D-37075 Gottingen, GermanyLeibniz Inst Primate Res, German Primate Ctr, D-37077 Gottingen, GermanyLeibniz Sci Campus Primate Cognit, D-37077 Gottingen, GermanyUniv Fed Sao Paulo, Inst Sci & Technol, BR-12231280 Sao Jose Dos Campos, SP, BrazilEuropean Union's Horizon 2020 Research and Innovation Programme: 720270German Research Foundation: WI 4046/1-1National Institutes of Health/National Institute of Neurological Disorders and Stroke: 1R03NS096379FAPESP: 2016/08263-9EU Grant H2020-FETOPEN: RIA 686764Web of Scienc

An integrative, multiscale view on neural theories of consciousness.

How is conscious experience related to material brain processes? A variety of theories aiming to answer this age-old question have emerged from the recent surge in consciousness research, and some are now hotly debated. Although most researchers have so far focused on the development and validation of their preferred theory in relative isolation, this article, written by a group of scientists representing different theories, takes an alternative approach. Noting that various theories often try to explain different aspects or mechanistic levels of consciousness, we argue that the theories do not necessarily contradict each other. Instead, several of them may converge on fundamental neuronal mechanisms and be partly compatible and complementary, so that multiple theories can simultaneously contribute to our understanding. Here, we consider unifying, integration-oriented approaches that have so far been largely neglected, seeking to combine valuable elements from various theories

Hippocampus Leads Ventral Striatum in Replay of Place-Reward Information

While the brain is asleep, the hippocampus plays first fiddle in the orchestra of memory; spatial information is reactivated in the hippocampus shortly in advance of emotional memory traces in the vental striatum