Multimodal Grounding for Language Processing

This survey discusses how recent developments in multimodal processing facilitate conceptual grounding of language. We categorize the information flow in multimodal processing with respect to cognitive models of human information processing and analyze different methods for combining multimodal representations. Based on this methodological inventory, we discuss the benefit of multimodal grounding for a variety of language processing tasks and the challenges that arise. We particularly focus on multimodal grounding of verbs which play a crucial role for the compositional power of language.Comment: The paper has been published in the Proceedings of the 27 Conference of Computational Linguistics. Please refer to this version for citations: https://www.aclweb.org/anthology/papers/C/C18/C18-1197

Enlightened Romanticism: Mary Gartside’s colour theory in the age of Moses Harris, Goethe and George Field

The aim of this paper is to evaluate the work of Mary Gartside, a British female colour theorist, active in London between 1781 and 1808. She published three books between 1805 and 1808. In chronological and intellectual terms Gartside can cautiously be regarded an exemplary link between Moses Harris, who published a short but important theory of colour in the second half of the eighteenth century, and J.W. von Goethe’s highly influential Zur Farbenlehre, published in Germany in 1810. Gartside’s colour theory was published privately under the disguise of a traditional water colouring manual, illustrated with stunning abstract colour blots (see example above). Until well into the twentieth century, she remained the only woman known to have published a theory of colour. In contrast to Goethe and other colour theorists in the late 18th and early 19th century Gartside was less inclined to follow the anti-Newtonian attitudes of the Romantic movement

Language as a disruptive technology: Abstract concepts, embodiment and the flexible mind

A growing body of evidence suggests that cognition is embodied and grounded. Abstract concepts, though, remain a significant theoretical chal- lenge. A number of researchers have proposed that language makes an important contribution to our capacity to acquire and employ concepts, particularly abstract ones. In this essay, I critically examine this suggestion and ultimately defend a version of it. I argue that a successful account of how language augments cognition should emphasize its symbolic properties and incorporate a view of embodiment that recognizes the flexible, multi- modal and task-related nature of action, emotion and perception systems. On this view, language is an ontogenetically disruptive cognitive technology that expands our conceptual reach

Perceptual simulation in conceptual tasks

No abstract available

Aiming for Cognitive Equivalence – Mental Models as a Tertium Comparationis for Translation and Empirical Semantics

This paper introduces my concept of cognitive equivalence (cf. Mandelblit, 1997), an attempt to reconcile elements of Nida’s dynamic equivalence with recent innovations in cognitive linguistics and cognitive psychology, and building on the current focus on translators’ mental processes in translation studies (see e.g. Göpferich et al., 2009, Lewandowska-Tomaszczyk, 2010; Halverson, 2014). My approach shares its general impetus with Lewandowska-Tomaszczyk’s concept of re-conceptualization, but is independently derived from findings in cognitive linguistics and simulation theory (see e.g. Langacker, 2008; Feldman, 2006; Barsalou, 1999; Zwaan, 2004). Against this background, I propose a model of translation processing focused on the internal simulation of reader reception and the calibration of these simulations to achieve similarity between ST and TT impact. The concept of cognitive equivalence is exemplarily tested by exploring a conceptual / lexical field (MALE BALDNESS) through the way that English, German and Japanese lexical items in this field are linked to matching visual-conceptual representations by native speaker informants. The visual data gathered via this empirical method can be used to effectively triangulate the linguistic items involved, enabling an extra-linguistic comparison across languages. Results show that there is a reassuring level of interinformant agreement within languages, but that the conceptual domain for BALDNESS is linguistically structured in systematically different ways across languages. The findings are interpreted as strengthening the call for a cognition-focused, embodied approach to translation

Empiricism without Magic: Transformational Abstraction in Deep Convolutional Neural Networks

In artificial intelligence, recent research has demonstrated the remarkable potential of Deep Convolutional Neural Networks (DCNNs), which seem to exceed state-of-the-art performance in new domains weekly, especially on the sorts of very difficult perceptual discrimination tasks that skeptics thought would remain beyond the reach of artificial intelligence. However, it has proven difficult to explain why DCNNs perform so well. In philosophy of mind, empiricists have long suggested that complex cognition is based on information derived from sensory experience, often appealing to a faculty of abstraction. Rationalists have frequently complained, however, that empiricists never adequately explained how this faculty of abstraction actually works. In this paper, I tie these two questions together, to the mutual benefit of both disciplines. I argue that the architectural features that distinguish DCNNs from earlier neural networks allow them to implement a form of hierarchical processing that I call “transformational abstraction”. Transformational abstraction iteratively converts sensory-based representations of category exemplars into new formats that are increasingly tolerant to “nuisance variation” in input. Reflecting upon the way that DCNNs leverage a combination of linear and non-linear processing to efficiently accomplish this feat allows us to understand how the brain is capable of bi-directional travel between exemplars and abstractions, addressing longstanding problems in empiricist philosophy of mind. I end by considering the prospects for future research on DCNNs, arguing that rather than simply implementing 80s connectionism with more brute-force computation, transformational abstraction counts as a qualitatively distinct form of processing ripe with philosophical and psychological significance, because it is significantly better suited to depict the generic mechanism responsible for this important kind of psychological processing in the brain

Understanding analogical reasoning : viewpoints from psychology and related disciplines

Analogy and metaphor have a long history of study in linguistics, education, philosophy and psychology. Consensus over what analogy is or how analogy functions in language and thought, however, has been elusive. This paper, the first in a two part series, examines these various research traditions, attempting to bring out major lines of agreement over the role of analogy in individual human experience. As well as being a general literature review which may be helpful for newcomers to the study of analogy, this paper attempts to extract from these literatures existing theories, models and concepts which may be interesting or useful for computational studies of analogical reasoning

Perceptual content and the content of mental imagery

The aim of this paper is to argue that the phenomenal similarity between perceiving and visualizing can be explained by the similarity between the structure of the content of these two different mental states. And this puts important constraints on how we should think about perceptual content and the content of mental imager