Language with Vision: a Study on Grounded Word and Sentence Embeddings

Grounding language in vision is an active field of research seeking to construct cognitively plausible word and sentence representations by incorporating perceptual knowledge from vision into text-based representations. Despite many attempts at language grounding, achieving an optimal equilibrium between textual representations of the language and our embodied experiences remains an open field. Some common concerns are the following. Is visual grounding advantageous for abstract words, or is its effectiveness restricted to concrete words? What is the optimal way of bridging the gap between text and vision? To what extent is perceptual knowledge from images advantageous for acquiring high-quality embeddings? Leveraging the current advances in machine learning and natural language processing, the present study addresses these questions by proposing a simple yet very effective computational grounding model for pre-trained word embeddings. Our model effectively balances the interplay between language and vision by aligning textual embeddings with visual information while simultaneously preserving the distributional statistics that characterize word usage in text corpora. By applying a learned alignment, we are able to indirectly ground unseen words including abstract words. A series of evaluations on a range of behavioural datasets shows that visual grounding is beneficial not only for concrete words but also for abstract words, lending support to the indirect theory of abstract concepts. Moreover, our approach offers advantages for contextualized embeddings, such as those generated by BERT, but only when trained on corpora of modest, cognitively plausible sizes. Code and grounded embeddings for English are available at https://github.com/Hazel1994/Visually_Grounded_Word_Embeddings_2

Language and action in Broca’s area: Computational differentiation and cortical segregation

Actions have been proposed to follow hierarchical principles similar to those hypothesized for language syntax. These structural similarities are claimed to be reflected in the common involvement of certain neural populations of Broca’s area, in the Inferior Frontal Gyrus (IFG). In this position paper, we follow an influential hypothesis in linguistic theory to introduce the syntactic operation Merge and the corresponding motor/conceptual interfaces. We argue that actions hierarchies do not follow the same principles ruling language syntax. We propose that hierarchy in the action domain lies in predictive processing mechanisms mapping sensory inputs and statistical regularities of action-goal relationships. At the cortical level, distinct Broca’s subregions appear to support different types of computations across the two domains. We argue that anterior BA44 is a major hub for the implementation of the syntactic operation Merge. On the other hand, posterior BA44 is recruited in selecting premotor mental representations based on the information provided by contextual signals. This functional distinction is corroborated by a recent meta-analysis (Papitto, Friederici, & Zaccarella, 2020). We conclude by suggesting that action and language can meet only where the interfaces transfer abstract computations either to the external world or to the internal mental world

Mind out of matter: topics in the physical foundations of consciousness and cognition

This dissertation begins with an exploration of a brand of dual aspect monism and some problems deriving from the distinction between a first person and third person point of view. I continue with an outline of one way in which the conscious experience of the subject might arise from organisational properties of a material substrate. With this picture to hand, I first examine theoretical features at the level of brain organisation which may be required to support conscious experience and then discuss what bearing some actual attributes of biological brains might have on such experience. I conclude the first half of the dissertation with comments on information processing and with artificial neural networks meant to display simple varieties of the organisational features initially described abstractly.While the first half begins with a view of conscious experience and infers downwards in the organisational hierarchy to explore neural features suggested by the view, attention in the second half shifts towards analysing low level dynamical features of material substrates and inferring upwards to possible effects on experience. There is particular emphasis on clarifying the role of chaotic dynamics, and I discuss relationships between levels of description of a cognitive system and comment on issues of complexity, computability, and predictability before returning to the topic of representation which earlier played a central part in isolating features of brain organisation which may underlie conscious experience.Some themes run throughout the dissertation, including an emphasis on understanding experience from both the first person and the third person points of view and on analysing the latter at different levels of description. Other themes include a sustained effort to integrate the picture offered here with existing empirical data and to situate current problems in the philosophy of mind within the new framework, as well as an appeal to tools from mathematics, computer science, and cognitive science to complement the more standard philosophical repertoire

The Cognitive Functions of Language

Includes peer critique and author responses.This paper explores a variety of different versions of the thesis that natural language is involved in human thinking. It distinguishes amongst strong and weak forms of this thesis, dismissing some as implausibly strong and others as uninterestingly weak. Strong forms dismissed include the view that language is conceptually necessary for thought (endorsed by many philosophers) and the view that language is de facto the medium of all human conceptual thinking (endorsed by many philosophers and social scientists). Weak forms include the view that language is necessary for the acquisition of many human concepts, and the view that language can serve to scaffold human thought processes. The paper also discusses the thesis that language may be the medium of conscious propositional thinking, but argues that this cannot be its most fundamental cognitive role. The idea is then proposed that natural language is the medium for non-domain-specific thinking, serving to integrate the outputs of a variety of domain-specific conceptual faculties (or central-cognitive ‘quasi-modules’). Recent experimental evidence in support of this idea is reviewed, and the implications of the idea are discussed, especially for our conception of the architecture of human cognition. Finally, some further kinds of evidence which might serve to corroborate or refute the hypothesis are mentioned. The overall goal of the paper is to review a wide variety of accounts of the cognitive function of natural language, integrating a number of different kinds of evidence and theoretical consideration in order to propose and elaborate the most plausible candidate

Models, Simulations, and the Reduction of Complexity

Modern science is a model-building activity. But how are models contructed? How are they related to theories and data? How do they explain complex scientific phenomena, and which role do computer simulations play? To address these questions which are highly relevant to scientists as well as to philosophers of science, 8 leading natural, engineering and social scientists reflect upon their modeling work, and 8 philosophers provide a commentary

Change blindness: eradication of gestalt strategies

Arrays of eight, texture-defined rectangles were used as stimuli in a one-shot change blindness (CB) task where there was a 50% chance that one rectangle would change orientation between two successive presentations separated by an interval. CB was eliminated by cueing the target rectangle in the first stimulus, reduced by cueing in the interval and unaffected by cueing in the second presentation. This supports the idea that a representation was formed that persisted through the interval before being 'overwritten' by the second presentation (Landman et al, 2003 Vision Research 43149–164]. Another possibility is that participants used some kind of grouping or Gestalt strategy. To test this we changed the spatial position of the rectangles in the second presentation by shifting them along imaginary spokes (by ±1 degree) emanating from the central fixation point. There was no significant difference seen in performance between this and the standard task [F(1,4)=2.565, p=0.185]. This may suggest two things: (i) Gestalt grouping is not used as a strategy in these tasks, and (ii) it gives further weight to the argument that objects may be stored and retrieved from a pre-attentional store during this task

Models, Simulations, and the Reduction of Complexity

Modern science is a model-building activity. But how are models contructed? How are they related to theories and data? How do they explain complex scientific phenomena, and which role do computer simulations play? To address these questions which are highly relevant to scientists as well as to philosophers of science, 8 leading natural, engineering and social scientists reflect upon their modeling work, and 8 philosophers provide a commentary

Neurodemocracy: Self-Organization of the Embodied Mind

This thesis contributes to a better conceptual understanding of how self-organized control works. I begin by analyzing the control problem and its solution space. I argue that the two prominent solutions offered by classical cognitive science (centralized control with rich commands, e.g., the Fodorian central systems) and embodied cognitive science (distributed control with simple commands, such as the subsumption architecture by Rodney Brooks) are merely two positions in a two-dimensional solution space. I outline two alternative positions: one is distributed control with rich commands, defended by proponents of massive modularity hypothesis; the other is centralized control with simple commands. My goal is to develop a hybrid account that combines aspects of the second alternative position and that of the embodied cognitive science (i.e., centralized and distributed controls with simple commands). Before developing my account, I discuss the virtues and challenges of the first three. This discussion results in a set of criteria for successful neural control mechanisms. Then, I develop my account through analyzing neuroscientific models of decision-making and control with the theoretical lenses provided by formal decision and social choice theories. I contend that neural processes can be productively modeled as a collective of agents, and neural self-organization is analogous to democratic self-governance. In particular, I show that the basal ganglia, a set of subcortical structures, contribute to the production of coherent and intelligent behaviors through implementing “democratic" procedures. Unlike the Fodorian central system—which is a micro-managing “neural commander-in-chief”—the basal ganglia are a “central election commission.” They delegate control of habitual behaviors to other distributed control mechanisms. Yet, when novel problems arise, they engage and determine the result on the basis of simple information (the votes) from across the system with the principles of neurodemocracy, and control with simple commands of inhibition and disinhibition. By actively managing and taking advantage of the wisdom-of-the-crowd effect, these democratic processes enhance the intelligence and coherence of the mind’s final "collective" decisions. I end by defending this account from both philosophical and empirical criticisms and showing that it meets the criteria for successful solution